Mamba: From Intuition to Proof
How delta-gated state-space models challenge the Transformer — from the intuition of selective memory to the parallel-scan proof.
Efficient Hybrid Models
Long-context AI is hitting a quadratic wall.
I research the architectures that break through it — state-space models, linear attention, hybrids, and latent-reasoning loops that could make long-document AI affordable for hospitals, labs, and researchers who don't own a GPU cluster.
Every new token, replayed vs. remembered
Both models read the same sentence. Self-attention re-reads every previous token, every step. A selective SSM asks one question per token — worth remembering? — writes “keys” into its fixed-size state, and lets “the” pass straight through.
Transformer · KV-cache grows with every token
Selective SSM · fixed state — keeps meaning, skips filler
0
architectures mapped in the Atlas
0
published deep-dives on Towards AI
0
survey paper in preparation
0
independent research directions in progress
The problem
Attention pays for context twice — in compute and in memory.
Self-attention compares every token with every other token, so compute grows with the square of context length — and the key–value cache it keeps grows linearly on top of that. Drag the slider: the numbers below are computed live for a real model (Llama-3-8B, FP16) against a Mamba-2-class model whose recurrent state never grows at all.
—
token-pair comparisons (N²) per attention map
—
Transformer KV-cache (Llama-3-8B, FP16)
0.13 GB
SSM recurrent state — constant at any length
—
memory advantage at this context length
Inference memory vs. context length
KV-cache of Llama-3-8B vs. the fixed recurrent state of a Mamba-2-class 7B model. Hover for values; the marker tracks the slider.
Transformer KV-cache
SSM recurrent state
How these numbers are computed
KV-cache: 2 (K and V) × 32 layers × 8 KV heads × 128 head-dim × 2 bytes (FP16) = 128 KiB per token for Llama-3-8B with grouped-query attention. SSM state: ≈64 layers × 8192 inner-dim × 128 state-dim × 2 bytes ≈ 128 MiB, independent of context length. Attention scores: N² comparisons per layer-head pass.
And even with the compute…
Models lose the middle of long documents.
Even when a document fits in the context window, retrieval accuracy dips sharply when the relevant passage sits in the middle — the lost-in-the-middle failure documented by Liu et al. (2023). In clinical records, legal texts, and dense academic literature, the middle is where the evidence lives. My research plan measures exactly where this degradation begins, then tests whether selective state-space models can avoid it.
Retrieval accuracy vs. position of the relevant passage
Illustrative U-shaped curve after Liu et al. (2023), “Lost in the Middle: How Language Models Use Long Contexts.” Hover the points.
The pragmatic answer
Hybrids: keep a sliver of attention, make the rest linear.
Pure SSMs are cheap but fuzzy at verbatim recall; pure attention recalls perfectly but pays the quadratic tax on every layer. Production models now split the difference — a thin slice of real attention for precise, needle-in-haystack lookups, riding on a linear-state backbone that does the bulk of the work. Hover the stacks: orange layers are the only ones that keep a KV-cache.
≈8%
attention layers in Nemotron-H — the rest is Mamba-2 + FFN
6 / 52
attention layers in Nemotron 3 Nano 30B — 23 Mamba-2, 23 MoE
1 : 7
attention : Mamba ratio that Jamba shipped first
1M
token context window Nemotron 3 runs on this recipe
Where the attention actually is
Layer-by-layer composition, drawn to each model's real layer counts. The hybrid recipe: replace almost every attention layer with a constant-memory Mamba-2 layer, keep a few for high-fidelity recall.
Attention (keeps KV-cache)
Mamba-2 (constant state)
FFN (dense)
MoE (sparse experts)
Aug 2025 · Jet-Nemotron
“Gated DeltaNet achieves the best accuracy.”
NVIDIA's PostNAS ran six linear operators head-to-head — RWKV-7, RetNet, Mamba-2, GLA, DeltaNet, Gated DeltaNet — inside a frozen Transformer. Gated DeltaNet won; they built JetBlock on top of it.
My deep-dive: Inside Jet-Nemotron →
Dec 2025 · Nemotron 3
The frontier family ships on Mamba-2.
Four months later, the same lab bet its flagship open models — Nano, Super, and the 550B Ultra — on Mamba-2, not Gated DeltaNet, not JetBlock. What wins a 2B-scale search isn't what gets trusted at 550B.
My deep-dive: Nemotron 3's Mamba-MoE engine →Same lab. Same year. Two different winners. The field agrees on the hybrid recipe but not on the linear operator inside it — or where the attention layers belong, or whether any of it holds at the next order of magnitude. Nobody has a settled theory yet. That open question is precisely where my 24-month research plan and survey sit: put the operators under one notation, then measure — at kernel level — what the state actually remembers.
The third answer
Don't replace attention — give it a lower-resolution memory.
Hybrids delete most attention layers and hand the work to a recurrent state. DeepSeek-V4 takes a different route: keep attention in every layer, but let most of what it reads be compressed. Each layer attends over a short local window at full resolution and a long history at reduced resolution, concatenated into a single softmax. Two new layer kinds do this — CSA with light compression and a learned indexer that keeps only the top-scoring entries, and HCA with heavy compression and no indexer at all.
swa — local only
A causal sliding window and nothing else. Cheap bootstrap layers that resolve local structure; anything older is simply invisible to them.
csa — compressed sparse
Light compression, overlapping blocks, and a Lightning Indexer that ranks the pool and keeps the top-k. It ranks after the causal mask, so its selection is always a subset of what dense attention would have seen.
hca — heavily compressed
Aggressive non-overlapping compression, every closed block visible, no indexer to train. Only complete blocks are emitted — a partial one would summarise fewer tokens than its siblings and make the entries incomparable.
Why this matters for the hybrid debate. A Mamba-2 or KDA layer buys constant memory by compressing the past into a fixed-size state and giving up random access to it. A CSA or HCA layer compresses the past into fewer keys and keeps random access over them. Those are genuinely different bets about what long-context memory is for — which is exactly the comparison BitFuse was built to run, under identical ternary weights.
Two other levers
Spend fewer bits per weight, or fewer steps per answer.
Replacing attention with a fixed-size state is only one way to make long-context AI affordable. Two Atlas nodes attack the cost from completely different directions: 1-bit models shrink what a weight costs to store and multiply, and diffusion language models shrink how many forward passes an answer costs to produce. Both are in the survey; neither needs a bigger GPU.
Where these sit in the map: 1-bit / MatMul-free is a Phase-1 node, diffusion LLMs are Phase 3 — both are covered in the survey's coverage map.
Putting it together
Everything above upgrades a different part of the same block.
Four slots, rebuilt by different labs on separate timelines — and, until 2026, almost never in the same model.
Step back from the individual papers and something odd shows up. The 2017 Transformer block has four swappable slots — a token mixer, a channel mixer, the precision the weights are stored in, and the residual pathway that carries information between layers. NVIDIA's Mamba-2 hybrids, Moonshot's delta attention, DeepSeek's compressed attention and Qwen's Gated DeltaNet have all been rebuilding the first. Mixture-of-experts rebuilt the second. BitNet rebuilt the third. They are largely orthogonal upgrades, and the fourth slot went nine years untouched — then got three incompatible replacements in a single year.
The original block — and its four slots
Vaswani et al. (2017). Every model on this page is still this diagram with one or two slots replaced. Slot 4 — the residual stream and Add & Norm skeleton — went unchanged for nine years, which is why it is drawn dashed. It is no longer empty: three labs moved it in 2026, in three different directions.
Who upgraded which slot
| Model | 1 · Token mixer | 2 · Channel mixer | 3 · Precision | 4 · Residual | Slots left original |
|---|---|---|---|---|---|
| Transformer 2017 | Multi-head attention | Dense FFN | FP16/FP32 | Add & Norm | 4 of 4 |
| BitNet b1.58 2025 | Full attention + RoPE | Dense FFN (squared ReLU) | Ternary, 1.58-bit | Add & Norm | 3 of 4 |
| Nemotron-H / 3 2025 | Mamba-2 hybrid (6 of 52 attn; 1M ctx) | Sparse MoE (23 of 52 layers) | FP/BF16 | Add & Norm | 2 of 4 |
| Kimi Linear 2025 | KDA 3:1 hybrid (delta rule) | Sparse MoE (3B of 48B active) | FP/BF16 | Add & Norm | 2 of 4 |
| DeepSeek-V4 2026 | SWA + CSA + HCA (local window kept in every layer) | Sparse MoE (61 layers) | FP8/FP4 storage | mHC (manifold-constrained hyper-connections) | 0 of 4 |
| Qwen3.8-Flash-Next 2026 | GDN 3:1 QSA (fixed state + block-indexed sparse retrieval) | Sparse MoE (multimodal) | FP/BF16 | Gated Residual (4 branches, read/write gated) | 1 of 4 |
| Kimi K3 2026 | KDA + global MLA (joint language/vision context) | Sparse MoE | FP/BF16 | Attention Residuals (softmax over depth) | 1 of 4 |
| BitFuse this work · A–L | KDA · Mamba-2 · GDN+QSA · CSA/HCA (ablated one at a time) | Dense — MoE deferred to Phase 6 | Ternary, 1.58-bit | Five families swept — gated, GR, mHC, AttnRes | 1 of 4 |
Update · 2026 — the empty corner got occupied, from three sides at once. DeepSeek-V4 was first to move all four slots together: a three-way hybrid attention schedule, sparse MoE, FP8/FP4 storage, and manifold-constrained hyper-connections in place of plain Add & Norm. Within months Qwen3.8-Flash-Next and Kimi K3 did the same thing with different answers in every slot — Qwen with Gated DeltaNet plus block-indexed sparse retrieval and a four-branch gated residual, Kimi with delta attention plus global MLA and a residual that attends over depth. Between them the “can these compose?” question is settled at frontier scale and in the affirmative. But each answer arrives by changing everything simultaneously, so none of them can tell you which slot bought which gain — and none was tried at low precision. That is the gap BitFuse works in: the same components, one at a time, at 1.5B where an ablation is affordable.
Slot 4 is now a crowded slot, not an empty one — and that is the more interesting problem. Nine years with one fixed addition, then three incompatible replacements in a single year. Qwen widens the stream and gates the reads. DeepSeek widens it too but pins the mixing matrix to the doubly-stochastic manifold so it keeps behaving like an identity map. Kimi leaves the stream alone and instead lets a layer run softmax attention over the outputs of every layer beneath it. Those are three different mechanisms — widening, constraining, and reaching along depth — routinely lumped together as “better residuals.” BitFuse treats them as separate algorithmic families with separate predicted signatures, because a win for one implies nothing at all about the others.
One honest correction, because it changes the argument. It's tempting to say BitNet's FFN is “the old part” — but BitNet ternarizes every linear layer, FFN included. The slot it actually left untouched is the token mixer: BitNet b1.58 2B4T still runs full softmax attention with RoPE, and its own technical report names extending sequence length and “investigating efficient attention mechanisms suitable for low-bit models” as future work. So the gap isn't a stale FFN — it's that the cheapest weights in the field are still attached to the most expensive sequence mixer, and the FFN, while ternary, is still dense.
That is what makes the corner interesting rather than merely unbuilt. The shipped savings multiply against different bottlenecks — ternary weights cut memory and bandwidth per parameter, hybrid and compressed attention cut the cache that grows with context, MoE cuts how many parameters run per token. Whether they actually compose at low precision, or interfere, is still an experimental question: no frontier model has taken the ternary route, ternary experts may not specialize, and a compressed recurrent state may not survive 1.58-bit weights.
Slot 4: shut for nine years, then opened three ways at once
Every block above except the 2026 three still moves information between layers the way it did in 2017 — one addition, from the layer directly beneath:
$$ x_{l+1} = x_{l} + F(x_{l}) $$
That is a write, not a read. Layer 20 sees layer 19's output and nothing else; anything layer 4 computed has to survive sixteen rounds of being overwritten to still be there. In a full-precision network that is usually fine. Under ternary weights, every one of those sixteen transformations is lossy by construction — which is exactly why this slot becomes worth reopening at 1.58 bits and not before.
Attention residuals make the pathway a retrieval instead of an addition. A layer attends back over the earlier states in the stack and pulls what it needs — which is exactly what Kimi K3 shipped in 2026:
$$ x_{l+1} = x_{l} + F(x_{l}) + \mathrm{Attn}\!\left(x_{l},\, \{x_{0}, \dots, x_{l-1}\}\right) $$
The hypothesis: if ternary transformations shed information layer by layer, letting a layer read earlier states directly may recover some of it — at a cost, since those states have to be kept and attended over. Toggle slot 4 in the builder above to see the pathway change. Three shipped mechanisms now occupy this slot and they are not variants of one idea: DeepSeek's mHC and Qwen's Gated Residual widen the stream, and only Kimi's AttnRes retrieves over depth. BitFuse sweeps all of them as separate families at ternary precision, where none has been tried.
How I'm testing it — BitFuse See the operators in the Atlas
Update · Aug 2026 — Nemotron 3.5 Lightning. The bet hardened: NVIDIA's newest release is a 30B mixture-of-experts with just 3B active parameters, on the same interleaved Mamba-2 + MoE backbone with a few attention layers. The interesting part: its headline “up to 4× output speed” doesn't come from the backbone at all — it comes from decoding. Multi-token prediction was trained in (with a dedicated MTP-boosting phase), and two draft models — DSpark and DFlash — drive speculative decoding on top. The race below shows why that works. No technical report on arXiv yet; details from the NVIDIA announcement ↗. Multi-token prediction is a Phase-2 architecture in my Atlas — I wrote about it in Why Your LLM Should Be Guessing ↗.
Why Lightning is fast: guess four, check once
Both lanes decode the same sentence. Autoregression pays one full forward pass per token. Speculative decoding lets a tiny draft model propose four tokens, then the big hybrid verifies them all in a single pass — wrong guesses get thrown out and redrafted, and it still finishes ~3× sooner.
Verified one at a time — 1 token per pass
Drafted then verified — up to 4 tokens per pass
Research direction I · proposal
Can state-space models make long-context reasoning affordable?
A 24-month empirical program: map the exact degradation curve of Transformer long-context retrieval, then test whether Mamba-2-class architectures can match it at a fraction of the memory — with hardware-level profiling on dedicated HPC infrastructure, and falsifiable success criteria fixed in advance.
≤ 5 pts
target retrieval-F1 gap vs. Transformer baseline beyond 32K tokens
≥ 40%
target reduction in peak KV-cache memory
3
phases: baseline · build · validate
O(N)
the complexity claim to be verified at kernel level
Research direction II · proposal v2 · 26 August 2026
Ternary weights cost quality. The question is where the information goes instead.
A separate line of work, now at proposal v2. BitFuse is a staged, controlled research programme on Qwen2.5-1.5B: twelve model variants, each changing one thing at a time so every result stays attributable. v2 adds Qwen3.8-Flash-Next's Gated DeltaNet + sparse-attention retrieval to Phase 4, promotes residual topology to a five-family experimental axis, and schedules multi-token prediction and multimodal integration as Phase 5.
0
variants across six core models, two Phase-4 sets and two Phase-5 extensions
0%
parameter-count spread held across all variants, by deliberate tuning
0
research questions, each tied to a named pair before any run
Δq/ΔR
the metric: marginal quality per unit of resource, per allocation axis
What v2 adds. Phase 4 now has three heterogeneous retrieval families: KDA, DeepSeek's compressed-attention schedule, and Qwen3.8-Flash-Next's Gated DeltaNet + Qwen Sparse Attention — where a micro-block indexer selects which positions the sparse head reads. The residual arm expands to five distinct topologies (R0–R4), each with a declared capacity budget so comparisons stay clean. MTP and multimodal integration join as Phase 5, each reported on a separate comparison set so they cannot quietly contaminate the Phase 2–4 perplexity table.
Deliberately kept apart from the survey work. Direction I and the survey ask whether the sequence mixer can be replaced. This one leaves that open and asks a different question at a different precision. Different hypotheses, different baselines, no shared claims — and any result from one does not transfer to the other.
The Architecture Atlas
Fourteen architectures. Three phases. One story.
Every post-Transformer architecture answers a specific failure of the one before it. The Atlas maps that evolutionary chain from LSTM (1997) to recursive language models (2026) — each node with its motivation, key equation, complexity, and the paper behind it.
Phase 1
The Linear-Sequence Revolution
Replace the KV-cache with a fixed-size state — RNNs, 1-bit LLMs, linear attention, hybrids, and Mamba.
Phase 2
Latent Reasoning Loops
Spend adaptive test-time compute inside a latent state — TTT, multi-token prediction, HRM, TRM, recursive LMs.
Phase 3
Beyond Causal Autoregression
Abandon next-token prediction — energy-based Transformers, diffusion LLMs, JEPA, and world models.
Writing
Deep-dives, published in Towards AI.
Selected articles — the full list, mapped to Atlas nodes, is on the Writing page.
Gated Attention & DeltaNets
The missing link for long-context AI: fusing Mamba's forgetting with DeltaNet's precise memory writes.
Tiny Recursion Models
How a 7M-parameter network with recursion beats models a thousand times its size on hard reasoning puzzles.
In preparation
Beyond the KV-Cache: A Survey of Transformer-Alternative Language Model Architectures
A survey organizing the fourteen Atlas architectures into their three evolutionary phases, under a single unified state-update notation — in preparation for arXiv (cs.CL / cs.LG). The Atlas is its living companion — and every one of the fourteen gets its own published deep-dive first.
7 of 14 architectures already have a published deep-dive
See the coverage map →