Research direction II · proposal v2 · 26 August 2026

Ternary weights cost quality. The question is where the information goes instead.

BitFuse is a staged research programme on the Qwen2.5-1.5B architecture under BitNet-style ternary parameterization. Its core thesis: when the static parameter channel is aggressively compressed, sequence mixing, residual pathways and conditional computation stop being independent plug-ins and become alternative information carriers competing for the same freed budget. The code, the fairness audit, the instrumentation and the Azure runbook exist; what is missing is GPU hours.

Not a leaderboard exercise. Every model here is trained from scratch on a fixed 1B-token certified budget, so all of them are weak in absolute terms. That is expected and fine: the absolute perplexity is not the result — the differences between the models are. Which is why the interesting clause in the hypothesis is the last one: a quality improvement with no matching change in carrier statistics would falsify the explanation even if the number went up.

0

variants across the core six, two Phase-4 sets and two Phase-5 extensions

0

residual topologies treated as separate algorithms, not settings

0

research questions, each tied to a named pair before any run

0B

token certified budget for every primary comparison

What changed in v2. Three things. The Phase 4 comparison now includes Qwen3.8-Flash-Next's Gated DeltaNet + Qwen Sparse Attention alongside the existing KDA and DeepSeek-inspired families. Residual design is promoted from a single arm to a first-class experimental axis spanning five distinct topologies. And multi-token prediction and multimodal integration become Phase 5, with MoE pushed back to Phase 6 — so predictive and cross-modal mechanisms get studied before conditional capacity is reinvested.

The hypothesis, stated so it can lose

Ternarization flattens the network's internal representations. Where a full-precision layer has a few large activations carrying most of the signal, a ternary layer's activations are more uniform. That loss of contrast is the mechanism behind the quality drop. If so, mechanisms that restore contrast — a state-based attention with per-channel gating (KDA), a state-space mixer (Mamba-2), Qwen's fixed-size memory paired with block-indexed sparse retrieval, compressed long-range attention, a widened or manifold-constrained residual stream, or an attention over depth itself — should recover quality, and the recovery should be visible in the statistics, not just in the loss curve.

BitNet b1.58 constrains every weight to $ \{-1, 0, +1\} $ — about 1.58 bits each instead of sixteen, so a weight costs roughly a tenth of a byte to store. The published recipe leaves one slot conspicuously untouched: BitNet still runs full softmax attention with RoPE, and its own technical report names “investigating efficient attention mechanisms suitable for low-bit models” as future work. BitFuse is that investigation — and the v2 revision widens it, because the newest literature strengthens the original thesis rather than replacing it: Qwen, DeepSeek and Kimi have each shipped a different answer to “which dynamic carrier should do the work”, at full precision, without anyone testing whether the ranking survives ternarization.

Six carriers, one of them crippled

It helps to treat the network as a set of distinct information carriers, and to notice that ternarization is not symmetric across them. Ternarization is primarily a reduction in per-weight resolution, not a total loss of magnitude information — the model keeps a full-precision latent parameter and derives its ternary forward weight through a scale factor. So the useful question is not “how much accuracy is lost?” but “which channels become more valuable when static weight resolution is constrained?”

Weights — heavily degraded

Persistent learned knowledge, now at three levels. Instrumented per layer: ternary distribution, zero fraction, scale factors, quantization error, weight entropy — plus layer-wise quantization error and scale drift, new in v2.

Activations — untouched

Still continuous, still expressive. Magnitude, variance, contrast, effective rank. Not a control knob — normalization eats deliberate rescaling, so these are read as evidence, never tuned.

Attention — untouched and scalable

Which tokens interact and what gets retrieved. Entropy, max probability, top-k mass, output norm — and in v2, QSA indexer recall and coverage, selected-block mass, CSA/HCA visibility statistics.

Recurrent state — a fixed-size memory

What a KDA, Mamba-2 or Gated DeltaNet layer has absorbed. State norm, update ratio, cosine retention, and GDN/KDA retention-and-forgetting curves.

Residual — promoted to an axis

No longer one arm but five families. Input/update norm ratio, plus GR branch usage, mHC mixing entropy and doubly-stochastic deviation, and AttnRes depth weights.

Prediction & modality — new in v2

Two carriers that did not exist in v1. Auxiliary-token losses, acceptance rate and speculative speedup for MTP; cross-modal probe accuracy, image-token saliency and modality-conditioned norms for vision.

That asymmetry is the opening. A ternary transformation cannot express fine distinctions in its weights, but the dynamic carriers can still create sharp differences in what each token reads, remembers and preserves. If a low-precision network is short of contrast, those are where it can still be manufactured — and the organizing principle of the whole programme follows: compress static weight expressivity, measure where the information moves, then reinvest the freed budget in whichever mechanism delivers the most quality per unit of memory or compute.

The hypotheses, H1–H8

Eight claims, written before the runs.

v1 had three. The v2 expansion adds five more, because the residual programme, the predictive objective and the multimodal extension each make a claim the original three do not cover. Each one is scoped so a null result is informative rather than merely disappointing.

H1

Dynamic reallocation

Ternarization changes where useful information is carried. If the sequence mixer is also too restrictive, quality may fall further than necessary — richer dynamic computation may compensate for constrained weights.

H2

Dynamic-capacity compensation

A ternary model may want a different allocation of global, local, recurrent and compressed-attention capacity than a full-precision one. This has to be measured, not assumed.

H3

Residual preservation

Residual topology can preserve or re-route information that would otherwise be attenuated, overwritten or diluted by a deep sequence-mixing stack.

H4

Heterogeneous context roles

Local retrieval, compressed retrieval, fixed-size recurrent memory and precise global attention may divide the representational work more efficiently than any single mixer.

H5

Depth-wise information selection

Kimi-style Attention Residuals may help ternary models by letting each layer select which earlier representations deserve to contribute — turning depth into an additional dynamic channel.

H6

Topology-stable multi-stream residuals

DeepSeek mHC and Qwen Gated Residuals may preserve multiple pathways while avoiding the instability of unconstrained hyper-connections.

H7

Predictive compression and acceleration

MTP can make more of the same backbone useful per forward pass, potentially improving speculative-decoding acceptance and backbone quality without altering the carrier thesis.

H8

Modality as a carrier

Multimodal inputs test whether a ternarized backbone can preserve cross-modal information while leaning harder on dynamic representations, connectors and residual pathways.

The falsification rule, made concrete

Every mechanism has to predict where the information moves.

This is what separates an explanation from a coincidence, and it is the single most important methodological commitment on the page. Each mechanism enters with a prediction about which carriers should gain and which should give way. If quality improves and the carriers do not move as predicted, the explanation is treated as refuted — the gain is real but unexplained, which is a different and weaker claim.

The trade, in bytes

Why dynamic capacity has to be very good to be worth buying.

Before any experiment runs, the arithmetic already says something uncomfortable. Ternary weights are so cheap that every attention, residual or prediction-head parameter costs many backbone parameters — and an order of magnitude more again if it stays at higher precision, as an indexer or a residual gate has to. That is the bar every hypothesis on this page has to clear, and it is why BitFuse holds parameter count fixed to within 0.2%: otherwise the better architecture would just be the bigger one.

This control is a design space, not a result. The horizontal axis of the real experiment is this same split; the vertical axis — quality — is what the measurements have to supply.

The experiment

Twelve models. One class. One switch at a time.

All of them share the Qwen2.5-1.5B skeleton — 28 layers, hidden 1536, 12 query heads, 2 KV heads, SwiGLU, RMSNorm, RoPE, tied embeddings — and all of them are one implementation driven by config switches, not twelve model files. Divergent copies are how “identical except for X” quietly stops being true. New mechanisms enter as separate expansion sets, so the primary table never fills up with partially controlled comparisons.

	Set	Name	Sequence mixer	Residual topology	Primary question
A	core	`fp_baseline`	softmax, all layers · FP	standard	the full-precision reference
B	core	`bitnet`	softmax, all layers	standard	what ternarization costs
C	core	`bitnet_kda`	3 KDA : 1 global	standard	does KDA recover the ternary loss?
D	core	`bitnet_kda_attnres`	3 KDA : 1 global	channel-gated	do low-cost residual gates add anything?
E	core	`bitnet_hybrid`	24 Mamba-2 : 4 global (Nemotron-H)	standard	does Mamba-2 survive ternarization?
F	core	`bitnet_hybrid_attnres`	24 Mamba-2 : 4 global	channel-gated	do residuals help Mamba-2 too?
G	4-D	`bitnet_deepseek`	SWA + CSA + HCA	standard	does heterogeneous compressed attention help?
H	4-D	`bitnet_mamba_deepseek`	Mamba-2 + CSA + HCA	standard	does compressed attention complement recurrence?
I	4-Q	`bitnet_qwen_hybrid`	3 GDN : 1 QSA (Qwen3.8)	standard	does recurrent memory + selective retrieval fit ternarization?
J	4-Q	`bitnet_qwen_hybrid_res`	3 GDN : 1 QSA	family sweep R1–R4	which residual topology best supports GDN + QSA?
K	5-P	`bitnet_mtp`	best Phase-4 mixer	best topology	does predictive supervision improve quality or only serving?
L	5-M	`bitfuse_multimodal`	best Phase-4 mixer	best topology	can a ternary backbone carry multimodal information?

A→D

the chain

Each step changes exactly one component: full precision → ternary → KDA → attention residual. Anything that moves is attributable to the step that moved it.

E, F

the branch

E and F branch off B rather than continuing from C: they swap in a different linear-attention family under identical ternary weights. So C and E are rivals, not successive steps — C vs E asks which family survives ternarization better.

G–J

Phase 4 — two expansion sets, reported separately

4-D keeps the DeepSeek compressed-attention track; 4-Q adds Qwen's GDN + QSA. Each has its own comparison set and its own report, so the six-way core ablation cannot quietly become a ten-way one asking a different question.

K, L

Phase 5 — extensions of the winner, not rivals to it

Both branch off whichever dense architecture Phase 4 leaves standing. K changes the training objective and L changes the input space, so neither belongs in the Phase 2–4 perplexity table at all — they get their own outcomes.

Why the Mamba ratio is not 3:1. The KDA variants use 3 KDA : 1 global, following Kimi Linear. The Mamba-2 variants deliberately do not mirror that, because copying NVIDIA's architecture means copying their placement. Nemotron-H's published hybrid_override_pattern is 52 single-purpose layers — 24 Mamba-2, 4 attention, 24 MLP, so attention is only 7.7% of layers. A BitFuse block holds a mixer and an MLP, so counting mixers only gives 28 — exactly BitFuse's depth, a 1:1 mapping — and reading the run lengths puts attention at mixer indices 4, 10, 16, 22. Matching NVIDIA matters more here than matching Kimi.

Where the mechanisms come from

Six borrowed mechanisms, each with the experiment that tests it.

These are external literature inputs, used to motivate experiments. None of them is a claim that BitFuse has already reproduced the effect. What each card says is: this is what the source contributes, this is why it bears on the carrier thesis, and this is the specific comparison that would settle it here.

Qwen3.8-Flash-Next

Gated DeltaNet + Qwen Sparse Attention

GDN compresses history into a fixed-size recurrent state; QSA runs a lightweight indexer that aggregates tokens into micro-blocks and selects the important regions for sparse global attention. Qwen reports a 3:1 GDN:attention pattern.

This is the BitFuse reallocation question stated in someone else's architecture: how should representational work split between recurrent memory and selective retrieval?

test → ternary GDN + QSA vs ternary KDA, plain attention and DeepSeek compressed attention, matched parameters and tokens

DeepSeek

SWA + CSA + HCA

Separates local context, compressed selective context and heavily compressed global context into three distinct attention roles rather than one mixer.

Already implemented here, and it becomes the main comparator for QSA: does heterogeneous attention stay valuable after ternarization?

test → keep the existing SWA/CSA/HCA track, run it against Qwen GDN+QSA and the KDA/Mamba hybrids

Qwen

Gated Residual (GR)

The residual stream is widened to four branches with dynamic, content-dependent read gating and per-branch write control.

Adds multiple residual information channels without requiring a separate attention mechanism to do it.

test → replace the channel-gated residual with a parameter-matched GR branch; probe whether branch specialization tracks the carrier metrics

DeepSeek

mHC — manifold-constrained hyper-connections

Hyper-connections widen the residual stream and route information dynamically; mHC constrains the residual mixing matrix to the doubly-stochastic manifold, restoring identity-map behaviour and stabilizing training.

A principled multi-stream topology whose stability is enforced rather than hoped for — which is exactly what makes it a fair rival to GR.

test → mHC vs GR vs baseline at matched capacity; log residual mixing entropy and singular-value/norm behaviour

Kimi K3

Attention Residuals

AttnRes replaces fixed residual accumulation with input-dependent attention over preceding layer outputs. Kimi also pairs KDA with global MLA.

This makes depth itself an information-selection mechanism, which is the carrier hypothesis applied along an axis the other four families do not touch.

test → full or block AttnRes as a separate residual family; measure depth-wise contribution concentration, output norms, gradient distribution

Qwen & Kimi

Multi-token prediction, and native multimodality

MTP trains future-token prediction heads; Qwen reports multi-step training aimed at consistency with speculative decoding, and Kimi K3 retains an MTP layer. Both models are also multimodal — Kimi trains language and vision jointly in one shared context.

Together they ask whether the carrier framework survives a change of training objective and a change of input space.

test → Phase 5-A: 1/2/4 future-token heads on the best dense model · Phase 5-B: frozen vision encoder and projector first, earlier fusion only if the backbone stays stable

Phase 4-Q · new in v2

Qwen's split: one layer kind to remember, another to retrieve.

Qwen3.8-Flash-Next puts Gated DeltaNet in most layers and Qwen Sparse Attention in periodic global ones, and frames it plainly as “remember” plus “retrieve”: GDN continuously compresses history into a fixed-size state, while QSA's indexer first aggregates tokens into micro-blocks and then selects the important regions to attend over. That is what makes it a genuinely different comparator from the existing KDA setup — the retrieval side is not dense global attention but sparse, block-indexed retrieval.

0 : 1

published GDN : QSA placement, kept as the starting point

0

layer budget unchanged — 21 GDN, 7 QSA

FP

the indexer's own parameters, declared separately from the ternary budget

0

of Qwen3.8's MoE or multimodal stack copied into Phase 4

Implementation stance. Begin at the published 3:1 placement, keep the 28-layer budget and the existing parameter-matching framework, and run a ternary GDN path for the projection matrices wherever the quantization contract permits it. Architectural attribution comes first; aggressive kernel optimization can wait, because a fast implementation of an uninterpretable comparison is worth nothing.

The indexer is declared, not assumed. QSA's indexer parameters are separated from the main ternary projection budget and the precision is stated explicitly — the same reasoning that keeps DeepSeek's index_q/index_k out of the name-matched ternary conversion pass. Quantizing a ranking corrupts selection for a negligible memory saving, and a corrupted ranking would be indistinguishable from the mechanism simply not working.

Three families, one controlled axis. Experiment 4.1 puts KDA + global against GDN + QSA at identical token budget, parameter budget, optimizer and seed. Experiment 4.2 puts GDN + QSA against SWA + CSA + HCA at the same 1.5B scale. Then 4.3 sweeps attention share (DS-25/50/75), 4.4 sweeps placement with the attention count fixed, and 4.5 sweeps head and indexer budget with the structure fixed — because “how much retrieval capacity” and “where the retrieval layers sit” are two different questions and answering them together answers neither. And if GDN + QSA wins, the claim to make is about fixed-size memory plus selective retrieval under ternary weights — not about Qwen being better.

Phase 4-D

DeepSeek's two compressed-attention layers, adapted to 1.5B.

The newest addition to the experiment takes the hybrid-attention schedule from the DeepSeek-V4 technical report and adapts it to this scale. Three layer kinds: swa reads a local window only; csa adds a lightly compressed, indexer-selected pool; hca adds a heavily compressed one. Every compressed layer keeps its local branch — the two key/value sets are concatenated and read by a single softmax.

0

local sliding window, kept by every DeepSeek layer kind

m = 0

CSA compression — overlapping, top-64 by indexer

m′ = 0

HCA compression — non-overlapping, no indexer

0

of V4-Pro's MoE, hyper-connections, partial RoPE or FP8 copied

These layers are not “attention over fewer tokens.” A compressed path alone cannot resolve fine-grained local structure, so removing the local branch would change what the layer is for. What each one actually does is read a short local window at full resolution and a long context at reduced resolution, in one softmax:

          ┌── local SWA (full resolution, window w) ──┐
hidden → ─┤                                           ├→ one softmax → out
          └── compressed K/V (+ indexer for CSA) ─────┘

Causality is structural, not assumed. A compressed entry covering source positions $[\text{start}, \text{end})$ summarises token $\text{end}-1$, so it is made readable only from that position onwards — and the Lightning Indexer ranks after that mask, which is what guarantees its selection is always a subset of what dense causal attention would have seen.

Only complete blocks are emitted: a partial trailing block would summarise fewer tokens than its siblings and make the entries incomparable. Tests assert that perturbing the tail of a sequence leaves every earlier logit bit-identical, that swa alone cannot see distant tokens while csa/hca can, and that cached decoding matches a full forward pass exactly.

Explicitly not a V4-Pro reproduction. V4-Pro is 61 layers of MoE with manifold-constrained hyper-connections, shared K=V multi-query attention, partial RoPE, grouped low-rank output projections, FP8/FP4 storage and a 1024-wide indexer. None of that is copied into Phase 4. Mixing it in would make it impossible to attribute a result to the sequence-mixing structure, which is the only thing Phase 4 tests. The indexer's projections are even named index_q/index_k specifically so the name-matching ternary conversion pass does not claim them — quantizing a ranking would corrupt selection for a negligible memory saving. Hyper-connections are a separate question about the residual pathway, and get their own family in the Phase 4-C sweep rather than riding along here — see below.

Phase 4-C · promoted to a first-class axis in v2

Five answers to “what should a layer be allowed to read?”

Since 2017 the residual pathway has been a single fixed addition from the layer directly beneath — a write, not a read. Under ternary weights every one of those transformations is lossy by construction, which is why the slot becomes worth reopening at 1.58 bits and not before. v1 tested two candidates. v2 tests five, and treats them as different algorithmic families rather than settings of one idea — because the distinction is what makes attribution possible at all.

	Family	Information path	Core operation	Expected signal	Main risk
R0	standard residual	1 stream	`x + F(x)`	identity preservation	depth-wise dilution
R1	channel-gated (current)	1 stream	`F(x) + α(x)·x`	content-dependent skip strength	limited path diversity
R2	Qwen Gated Residual	4 branches	dynamic read gate + per-branch write control	parallel pathways, selective preservation	routing and parameter overhead
R3	DeepSeek mHC	4+ branches	pre/post routing + doubly-stochastic mixing	stable multi-stream transport, identity-like	projection cost, implementation complexity
R4	Kimi Attention Residuals	depth-indexed	softmax over prior layer outputs	selective depth-wise aggregation	memory/communication cost; may need block approximation

R1 and R4 — reach. The current channel-gated residual modulates a single skip path at 3072 full-precision parameters per layer, initialized to zero so the module starts as a plain fixed residual and cannot destabilise early training. Kimi's AttnRes goes much further and makes the pathway a retrieval:

$$ x_{l+1} = x_{l} + F(x_{l}) + \mathrm{Attn}\!\left(x_{l},\, \{x_{0}, \dots, x_{l-1}\}\right) $$

The full gated matrix variant of R1 would add ~66M parameters (+4.1%) and make that model the biggest one, confounding the comparison it exists to run — which is precisely the trap the parameter-control rule below exists to catch.

R2 and R3 — width. Both widen the residual state, but they differ in how information is routed and constrained, so a win for one implies nothing about the other. mHC keeps the mixing matrix on the doubly-stochastic manifold:

$$ x_{l+1} = \sum_{k} \alpha_{k}^{(l)} \, x_{l-k} + \beta^{(l)} F(x_{l}), \qquad A \in \mathcal{DS} $$

The constraint is the contribution, not an implementation detail: it restores identity-like signal behaviour and keeps the recombination from drifting into an unstable scaling regime as depth grows. If mHC beats GR, the result to report is that constrained multi-stream routing trades stability against expressivity better than unconstrained gating does.

Why this comparison is scientifically useful. Widening the stream and selecting along depth are orthogonal moves — R4 expands what a layer can reach along depth, R2 and R3 expand how many parallel routes exist, and neither substitutes for the other. R1 stays in as the necessary low-complexity control, because the question “is multi-stream complexity actually required under ternarization?” has to have an answer. And a residual win only means anything mechanistically if the carrier metrics show the predicted change in norm, contrast, path utilization, depth contribution or branch specialization. To avoid confounding attention with topology, the sweep freezes the best two Phase-4 mixers and changes only the residual design.

Parameter control rule. Every residual family carries a declared capacity budget. Where a family adds parameters, the compensating reduction is solved in non-residual dimensions before the certified run. A comparison that pairs a more expressive residual with a larger total parameter count is a secondary result, not the primary causal one. The diagnostic is a residual carrier profile — branch norm share, residual update/input ratio, gate entropy, cross-layer cosine retention — and the prediction is not “more residual capacity is better” but that the best topology preserves high-value information without letting redundant or unstable pathways dominate.

Phase 4-D · RQ7, the capacity question made testable

How much dynamic retrieval is worth buying at 1.58 bits?

Not an assumption baked into an architecture, but a sweep. Ternarization removes fine-grained expressivity from the weights, so a ternary model may benefit from more dynamic sequence-mixing capacity than a full-precision one would. Three axes, moved one at a time, because changing two at once makes the result uninterpretable — and the v2 addition is that placement and indexer budget are now swept separately too, since “how much retrieval” and “where it sits” are different questions.

Axis 1 · share

DS-25 / DS-50 / DS-75

What fraction of layers do sequence mixing by attention rather than by recurrence. Attention is placed evenly, never clustered, so “how much attention” does not get confounded with “where the attention is”.

Axis 2 · depth

Experiment A — layer count

The number of attention layers moves; head configuration is frozen. Isolates “more attention layers” from “wider attention”.

Axis 3 · width

Experiment B — head count

Head count moves with the layer count fixed, and head_dim is pinned to hidden_size / heads — so total attention width and its parameter count stay constant. That is what separates attention diversity from attention capacity.

Axis 4 · indexer budget, new in v2

Experiment C — placement, then retrieval capacity

Where the dynamic retrieval layers occur with their count fixed, then how much indexer and head capacity they get with the structure fixed. That answers whether retrieval capacity or placement is the binding constraint — the question QSA's micro-block indexer makes newly worth asking.

Not an axis here

Residual topology, and MoE

Both change something other than sequence mixing — the residual pathway is its own Phase 4-C sweep, and MoE changes active parameters plus routing and waits until Phase 6. Folding either into this sweep would confound it. Kimi, Qwen and DeepSeek are all MoE at full scale; only their sequence mixers and residual topologies are borrowed here, never their expert routing.

Experiment B varies head geometry on purpose, which the fairness rules otherwise forbid. That needs an explicit opt-in — --phase4 --allow-attention-budget — and the differences are then recorded as declared, not waived. The comparison is fair within the sweep, which is exactly why Phase 4 reports on its own variant set.

What each comparison answers

Eleven questions, each tied to a specific pair.

Written down before the runs, so a result cannot be retrofitted to a question that happens to have been answered.

RQ1What is lost by ternarization at 1.5B scale?A vs B

RQ2Does KDA recover the ternary loss?B vs C

RQ3Does Mamba-2 provide a stronger recurrent alternative?C vs E

RQ4Do residual mechanisms recover information beyond what the mixer already carries?same-mixer ablations

RQ5Which residual family is best — channel-gated, Qwen GR, DeepSeek mHC, or Kimi AttnRes?R1–R4 sweep

RQ6Does Qwen GDN + QSA outperform the current KDA and DeepSeek hybrids under ternarization?C/G vs I

RQ7How much dynamic retrieval capacity is optimal under ternary weights?DS-25/50/75 + sweeps

RQ8Does the information-carrier hypothesis actually predict the quality differences?all carriers

RQ9Does MTP improve the same backbone, rather than merely serving it faster?K vs its base

RQ10Can a ternarized backbone retain multimodal information?L + cross-modal probes

RQ11Where should ternary memory savings be reinvested — dynamic context, residual topology, prediction heads, or MoE?Pareto, Phases 4–6

RQ8 is the one that distinguishes explanation from coincidence, RQ5 and RQ6 are what the v2 expansion exists to settle, and RQ11 is the question the other ten are instruments for.

RQ11, drawn

Five competing claims on the same freed memory, and a phase that owns each decision. Drag the split; the numbers are structural, not measured.

Phase 5-A · new in v2

Multi-token prediction, treated as two experiments rather than one.

Qwen reports MTP as a native component of its Next/3.8 lineage, using multi-step training to keep training consistent with speculative decoding while also improving the backbone; Kimi K3 retains an MTP layer too. BitFuse therefore treats it as both a training-objective experiment and an inference-efficiency experiment — never as an ordinary architecture-only ablation, because those two outcomes can move in opposite directions and merging them would hide it.

	Design	Primary outcome	Secondary outcome
MTP-1	no MTP — the reference	PPL / benchmark	—
MTP-2	predict 2 future tokens	backbone loss + acceptance rate	decode throughput
MTP-3	predict 4 future tokens	backbone loss + acceptance rate	training overhead
MTP-4	multi-step consistency schedule	speculative-decoding acceptance	stability / calibration
MTP-5	MTP on the Qwen-vs-DeepSeek best mixer	architecture × prediction interaction	Pareto efficiency

It has to be measured on a real serving engine. Acceptance rate and end-to-end throughput are the practical motivation for MTP, so they need an engine capable of speculative decoding rather than a synthetic estimate. And the report keeps language-model quality strictly separate from serving acceleration — otherwise a faster decoder gets mistaken for better training, which is the most likely way this particular experiment could produce a misleading headline.

Phase 5-B · new in v2

Does the carrier thesis survive leaving language?

Qwen3.8-Flash-Next is multimodal and Kimi K3 trains language and vision jointly in one shared backbone and context. That motivates a broader test — but the first target here is deliberately the smallest experiment that can answer whether a ternarized language backbone preserves multimodal signal at all. Four staged gates, each of which has to pass before the next is worth running.

M1adapter-first

Frozen vision encoder + trainable projector + ternary LM

Image–text instruction and captioning. Gate: does the backbone train stably and retain its text quality? If it does not, nothing downstream is interpretable.

gate · backbone stability + text-quality retention

M2

Partially trainable vision adapter + ternary LM

Cross-modal reasoning. Gate: does information transfer improve without destabilizing the language model?

gate · transfer gain without LM regression

M3

Early-fusion multimodal tokens

Joint context modelling. Gate: can the hybrid attention and residual topology actually route visual and textual information together, rather than one crowding out the other?

gate · joint routing evidence

M4

Best Phase-4 architecture, multimodal

The full controlled comparison. Gate: does the architecture that won on long-context language remain the winner cross-modally? A “no” here would be one of the more interesting results in the programme.

gate · does the ranking transfer?

The vision encoder is reported separately, always. Its parameter count never enters the language-backbone comparison, and it stays frozen initially — because the BitFuse question is how the ternarized language backbone redistributes information when the input space gets richer, not how much a good vision tower is worth. If multimodal performance only improves alongside a large vision-side budget, that improvement must not be attributed to the ternary backbone without a matched control. Full end-to-end vision–language training is a later extension, and only if the controlled adapter experiment succeeds first.

What “controlled” means here, enforced in code

Every model is randomly initialized. The pretrained Qwen checkpoint is used only to confirm the architecture and tokenizer load correctly — never as a starting point. Held identical across all runs: tokenizer, dataset and revision, data ordering, token budget, sequence length, optimizer, warmup, gradient clipping, validation set, seed, checkpoint policy, evaluation protocol, hardware class.

That is not enforced by discipline. A fairness audit loads each variant's run manifest and compares every controlled field against an allow-list of the switches that are the experiment; any mismatch marks the whole comparison NON-CONTROLLED_EXPERIMENT and the reporting CLI exits non-zero, so an invalid comparison cannot pass silently in automation. Comparing the full model config against an allow-list — rather than a hand-maintained list of things to check — is what catches a hidden change like rms_norm_eps drifting on one variant.

One difference is deliberate and declared. The ternary variants use BitNet's two-stage LR and weight-decay schedule, part of the published b1.58 recipe, while fp_baseline keeps Qwen's plain cosine. The audit records that explicitly in a declared_differences list instead of waiving it silently — and only schedule fields may be declared at all. Declaring an unequal token budget raises, because that is not a recipe difference, it is a different experiment.

Parameter count is held to within 0.2% for primary comparisons, which took deliberate tuning: KDA has no grouped-query KV saving and naive settings inflated it by 12%, a Mamba-2 mixer has an entirely different footprint, and the residual's full gated mode would have added ~66M parameters (+4.1%) — which is why the default is channel_gated at 3072 parameters per layer. Without that, “KDA is better” could just be “KDA is bigger”. Secondary matched-compute comparisons may relax the parameter rule, but they have to be labelled as such.

Three rules are new in v2, and each one exists because a v2 mechanism would otherwise contaminate the primary table. MTP changes the training objective, so it is a separate Phase 5 extension set and is never merged into the Phase 2–4 perplexity comparison. Multimodal runs report vision encoder and connector parameters separately, with language-backbone comparisons kept matched wherever that is possible at all. And every residual family arrives with a declared capacity budget, paid for outside the residual dimensions before the certified run.

Everything else holds as before: the same FineWeb-Edu revision, split, tokenization, shuffle seed and packing policy; the same optimizer family and schedule unless the experiment is an optimization study; the same precision, CUDA stack, kernel policy and evaluation environment; the same fixed cached validation set and cadence. Fake-quantized ternary measurements stay separate from genuine low-bit kernel measurements. Provenance recorded per run: git SHA, config hash, data revision, hardware, token count, checkpoint lineage, kernel versions, and every eviction and resume.

What this is not

Not a speed claim. Ternary weights here are simulated (fake-quantized), which is slower than fp16, not faster. Efficiency reports are labelled algorithmic vs kernel_optimized and the two are never mixed.
Not a state-of-the-art attempt. Optimizing for the best number would defeat the purpose; each architectural change has to stay attributable.
Not MoE — yet. Deferred to Phase 6, behind MTP and multimodality, precisely to preserve attribution: learn which dense dynamic mechanisms are worth having before asking whether ternary savings are better spent on experts.
Not a reproduction of anyone's frontier model. Mechanisms are borrowed from Qwen3.8, DeepSeek-V4 and Kimi K3 one family at a time and never together. Their expert routing, their FP8/FP4 storage and their full multimodal stacks stay out.
Not a claim that these effects are already reproduced. Everything attributed to an external paper is a literature input motivating an experiment, and is labelled as a proposal until BitFuse has run it.

Two bugs that argue for the whole methodology

Both were in the KDA kernel, both produced plausible-looking output, and neither was visible from a loss curve. The state decay was applied to the wrong axis — the state is $[d_v, d_k]$ and the gate indexes $d_k$. And the intra-chunk term needs $\exp(c_t - c_s)$ over cumulative log-decay; computing it as k / exp(c) overflows for long chunks or steep decay, so it is now split around the chunk midpoint to keep both exponential factors bounded by 1.

A chunk-parallel training kernel is asserted equal to an explicit recurrent reference to 2e-4 across chunk sizes, sequence lengths, initial states and streaming continuation. Without that reference to diff against, both bugs would have silently degraded models C and D and been misread as “KDA doesn't help.”

A third one is worth naming because it is invisible by construction: overriding _init_weights without calling super() leaves the RoPE inv_freq buffers zero-filled after from_pretrained, because they are non-persistent and absent from checkpoints. A resumed model then produces different hidden states from the one that was saved — identical weights, no error. It surfaced only as a save/load roundtrip test failing at ~1e-3. Every resumed training run would have been silently corrupt.

Sequencing

Each phase is a gate, not a milestone.

GPU time is the dominant cost, so the campaign is split into phases launched independently. Only continue if the previous phase looks right — and the later scripts refuse to start if an earlier phase's artifacts are missing or were trained at a different token budget, so an unusable comparison cannot be discovered after paying for it.

Localsynthetic

Do all the architectural switches execute at all?

Reduced models, synthetic data, one step long — it produces no comparable numbers. It exists to fail cheaply: a mis-wired mixer found on a laptop costs nothing, while the same bug found on a provisioned GPU VM costs real money. Runs on any CUDA GPU, auto-detected.

gate · every switch executes correctly

Phase 0no GPU spend

Engineering validation

Reference load, forward/backward/step, strict ternarity, KDA equivalence against an explicit recurrent reference, checkpoint round-trip.

gate · code trusted enough to spend GPU money on

Phase 110M tokens

One variant per family, on real GPUs, on disposable infrastructure

FP, ternary KDA and ternary Mamba-2 at 10M disposable tokens, then the resource group is destroyed so nothing keeps billing — teardown downloads the evidence first and refuses to delete anything if the download is empty. This is also the gate for the carrier instrumentation, because a diagnostic problem and an architecture problem must never be debugged at the same time.

gate · real kernels and instrumentation work

Phase 21B tokens

FP + BitNet + KDA at full budget

A is retrained here rather than reusing Phase 1's throwaway run, because the fairness audit needs a baseline recorded under the identical contract.

gate · certified evidence for ternarization and KDA

Phase 31B tokens

The residual arms, then the certified six-way core report

D, E and F trained and evaluated, and the whole core six reported together.

gate · core fairness audit passes

Phase 4A100M → 1B

Qwen GDN + QSA

Pilot at 100M to establish it is numerically sound, then a finalist run at the certified budget.

gate · numerically and causally interpretable

Phase 4B100M → 1B

DeepSeek SWA + CSA + HCA, and the Mamba variant

The existing compressed-attention track, kept intact as the main comparator for QSA.

gate · the DeepSeek comparison stays controlled

Phase 4C100M screen

The residual family sweep — R1 / R2 / R3 / R4

Best two Phase-4 mixers frozen, topology the only thing moving. Screened at 100M, finalists promoted to the certified budget.

gate · best topology identified at matched capacity

Phase 4Dtargeted

Attention share, placement, head and indexer budget

DS-25/50/75, then placement with the count fixed, then capacity with the structure fixed.

gate · optimal dynamic-capacity allocation identified

Phase 5A100M → 1B

Multi-token prediction on the best dense architecture

1/2/4 heads plus the multi-step consistency schedule, measured on a speculative-decoding engine.

gate · quality and speculative-decoding evidence, reported apart

Phase 5Badapter-first

Multimodal integration

M1 through M4, expanding only after each gate — vision parameters always reported separately.

gate · cross-modal evidence without backbone confounding

Phase 6deferred

BitFuse-MoE — conditional capacity, last

Only after the dense architecture is frozen. Dense-width against expert-count at roughly equal active compute, with routed and shared expert parameters kept separate from the backbone in the capacity ledger.

gate · the conditional-capacity reinvestment decision

The causal order is the point of the sequence, not the schedule: A→B isolates ternarization; B→C and B→E test two rival recurrent compensations; C/E→residual isolates topology; B/E→Phase 4 tests heterogeneous context allocation; the best mixer feeds the residual sweep; the best dense architecture feeds MTP and multimodal; and only the best overall dense architecture feeds Phase 6.

What gets measured

The headline number is not accuracy. It is marginal return per unit of resource, computed separately for each allocation axis:

$$ \mathrm{Eff}_{\text{attn}} = \frac{\Delta q}{\Delta R_{\text{attn}}}, \qquad \mathrm{Eff}_{\text{ternary}} = \frac{\Delta q}{\Delta R_{\text{ternary}}} $$

where $q$ is task quality and $R$ is spent resource — weight memory, KV-cache, latency, training FLOPs, or energy where it can be measured honestly.

Alongside that, every run is instrumented internally, because a result without a mechanism is not worth much: ternary distribution and zero fraction per layer, activation magnitude, variance, contrast and effective rank, attention entropy, concentration, span and head diversity, recurrent-state occupancy, and how much earlier-layer information the residual pathway is actually carrying. The instrumentation is emitted at three levels — light, standard, full — and Phase 1's job includes confirming all three hold finite values before anything commits to a 1B-token run.

v2 adds diagnostics for the new mechanisms, each chosen so its mechanism could fail visibly rather than quietly. Weights gain layer-wise quantization error and scale drift. Attention gains QSA indexer recall and coverage, selected-block mass, and CSA/HCA visibility statistics. Recurrent state gains GDN and KDA retention-and-forgetting curves. Residuals gain GR branch usage, mHC mixing entropy and deviation from double stochasticity, and AttnRes depth weights. MTP gains auxiliary-token losses, token acceptance rate and speculative-decoding speedup. Multimodal runs gain cross-modal retrieval and probe accuracy, image-token saliency, and modality-conditioned norm and entropy profiles.

Four outcomes, all of them informative

Ternary scaling wins. Dynamic capacity is not an efficient substitute for parameters at low precision — which retires a plausible idea and tightens the case for current practice.
Allocation wins. A smaller ternary backbone with richer retrieval, residual or predictive machinery beats a larger one at matched resources, and low-bit scaling has been leaving quality on the table.
They are complementary. Both axes help, and the useful contribution becomes the shape of the joint frontier.
Allocation is task-dependent. Short context favors weights, long context favors selective retrieval, reasoning favors conditional capacity — the most interesting outcome, because it means there is no single correct low-bit architecture, only a correct allocation for a workload.

What counts as a strong result

A strong result is emphatically not “the newest architecture wins.” It is a controlled trade-off that survives matched-budget comparison and has a plausible mechanistic explanation behind it. So the claims are pre-committed with their interpretations attached:

If Qwen GDN + QSA wins, the claim is about the usefulness of fixed-size memory plus selective retrieval under ternary weights — not about Qwen being superior.
If mHC beats Gated Residual, the result is that constrained multi-stream routing offers a better stability/expressivity trade than unconstrained gating.
If AttnRes wins, the contribution is evidence that depth-wise information selection is a carrier separate from parallel residual streams.
If MTP improves serving but not validation loss, that is still a meaningful efficiency result, and the paper labels it as exactly that.
If multimodal performance improves only with a large vision-side budget, the result is not attributed to the ternary backbone without a matched control.
Any fairness-audit failure marks the corresponding delta NON-CONTROLLED_EXPERIMENT and removes it from the primary causal claims entirely.

Working thesis

Extremely low-precision models should probably not be scaled by uniformly adding more low-precision parameters. Because ternarization redistributes where representational capacity actually lives, the more efficient strategy may be to spend the budget on the mechanisms that select, route, preserve and predict information — recurrent memory, selective retrieval, residual topology, prediction heads, and eventually conditional computation — with ternary weights providing the cheap substrate underneath. The shift under test is from scaling the weights to allocating the information pathways.

Status: proposal v2, dated 26 August 2026. The core six variants, the KDA equivalence tests, the DeepSeek attention paths, the fairness audit, the carrier instrumentation and the phased Azure runbook all exist and pass. The Phase 4-Q, 4-C, 5-A and 5-B sets are specified here and not yet implemented, and are labelled as proposals throughout. Nothing on this page is a result. If you work on low-bit training, linear-attention kernels, residual topology or capacity-matched evaluation and want to argue with the design before compute gets spent on it — that is still the most useful thing anyone could do. Reach me via Medium or GitHub.