ML//transformer//attention//linear attention//Gated DeltaNet

Gated DeltaNet is a linear-attention layer that updates its fixed-size state with the delta rule (it overwrites what the memory holds for the new key instead of only adding to it) and with a gate that lets old content decay, and it is used in hybrid language models as the cheap layer between full-attention layers. It was proposed by Yang, Kautz and Hatamizadeh at ICLR 2025, and it is the linear layer of the Qwen3.6 models.

Gated DeltaNet is a linear-attention layer that updates its fixed-size state with the delta rule (it overwrites what the memory holds for the new key instead of only adding to it) and with a gate that lets old content decay, and it is used in hybrid language models as the cheap layer between full-attention layers. It was proposed by Yang, Kautz and Hatamizadeh at ICLR 2025, and it is the linear layer of the Qwen3.6 models.

In a GGUF file these layers show up as tensors named ssm_* (ssm_conv1d, ssm_alpha, ssm_beta, ssm_out) beside attn_qkv and attn_gate, and the metadata key full_attention_interval = 4 says that every fourth layer is ordinary attention instead. Qwen3.6 35B-A3B has 30 Gated DeltaNet layers and 10 attention layers; the dense Qwen3.8 27B has 48 and 16.

Only the attention layers keep a growing KV cache, so the cache of these models is a quarter of what a pure transformer of the same shape would need: 20 KB per token for Qwen3.6 35B-A3B in 16-bit, against 80 KB if all 40 layers kept one.

The delta rule is what separates it from plain linear attention: writing a new value for a key replaces the old association instead of piling on top of it, which keeps the fixed state from filling with stale mixtures.

Each layer's state has a fixed size (about 2 MB per layer in Qwen3.6), so it does not grow with the conversation, and a server that saves a conversation's state must save it along with the KV cache.