DeepSeek V4's 1M-Token Context Uses 10% of V3's KV Cache

DeepSeek V4 runs a one-million-token context window using only 10% of its predecessor’s KV cache — not through hardware upgrades, but through three architectural techniques introduced in the last six months. Gemma 4 achieves 128K context on a mobile device using the same core insight. The KV cache is no longer a scaling problem. It is a design problem.

Why the KV Cache Is the Real Bottleneck

In transformer inference, the Key-Value cache is the dominant memory cost at long context. Every token you attend to requires storing its key and value tensors in GPU memory — and that memory grows linearly with context length. At one million tokens, standard attention with full KV storage becomes practically intractable on anything except hyperscaler clusters. This is why long-context capability has historically required either massive hardware budgets or compromised quality through chunked attention.

The Sebastian Raschka analysis of 45+ recent open-weight models reveals a convergent pattern: every frontier model released since late 2025 attacks this same bottleneck through some combination of KV compression, KV sharing, or attention sparsification. The era of the standard transformer as a baseline is effectively over for agentic AI applications that need to hold large contexts — entire codebases, document corpora, or multi-turn session histories.

Three Techniques Worth Understanding

KV Sharing in Gemma 4: Google’s April 2026 Gemma 4 family uses cross-layer KV reuse — the last n decoder layers share the Key and Value tensors from the final non-shared layer of the same attention type, rather than computing their own. At 10 shared layers out of 30 total, KV cache memory drops approximately 33%. On-device, this shrinks the cache from gigabytes to megabytes, enabling 128K or more context on mobile hardware. Quality impact: minimal per Google’s ablations.

Gemma 4 also pairs this with a 5:1 hybrid of sliding-window and global attention layers — five local layers attend to a 512-token window, one global layer attends to the full context. Combined with Proportional RoPE on global layers (which avoids the attention-sink problem at extreme distances), the architecture delivers full-length coherence without running full attention on every layer.

CSA and HCA in DeepSeek V4: DeepSeek’s April 2026 V4 models take compression further. Compressed Sparse Attention (CSA) pools every four consecutive KV tokens into a single compressed entry using learned softmax-gated weights, then a Lightning Indexer selects the top-k compressed blocks per query. At m=4, KV cache drops 4×. Heavily Compressed Attention (HCA) at m=128 reduces KV memory by 128×. DeepSeek V4 interleaves CSA and HCA layers — local detail preserved by CSA, long-range structure handled by HCA — stacked on top of the MLA (Multi-Head Latent Attention) from V3, which independently compresses KV tensors into low-rank latents.

Net result: at one million tokens, DeepSeek V4 Pro requires 27% of V3’s inference FLOPs and 10% of V3’s KV cache. That is not a modest optimization — it is an architectural discontinuity.

mHC Residuals: Both model families move away from standard residual connections. DeepSeek V4 replaces them with manifold-constrained hyper-connections (mHC), which expand the residual stream width by 4× while constraining the residual mixing matrix to the Birkhoff Polytope (doubly stochastic entries) via Sinkhorn-Knopp iteration. The constraint preserves signal magnitude regardless of depth — eliminating the catastrophic gain explosion that unconstrained hyper-connections produce — while enabling richer inter-layer context routing. Training overhead: 6.7% above standard residuals.

Source: Ahead of AI

Content created with AI assistance and reviewed for accuracy.