DeepSeek V4 Compressed Attention: How the KV-Cache Shrinks to Just 2%
April 29, 2026 · Independent DeepSeek Resource Hub
DeepSeek V4 keeps a 1 million-token context window while using roughly 2% of the KV-cache memory a standard transformer would need. The trick is a new architecture called Compressed Attention — a stack of CSA, HCA and low-rank projections that compresses along the sequence dimension instead of the head dimension.
TL;DR — DeepSeek V4 Compressed Attention in 60 seconds
• Problem: KV-cache memory grows linearly with sequence length and explodes at 1M tokens.
• Fix: Compress along the sequence dimension instead of the head dimension.
• CSA: Groups of ~4 tokens merged into one KV entry with data-dependent weighting and overlapping windows.
• HCA: Up to 128 tokens compressed into one entry — a cheap global summary.
• Low-rank: Queries shrunk to ~15.8% and output projections to ~14.3% of original size.
• Hybrid stack: HCA early, mixed middle, full attention at the final layer for precision.
• Result: ~2% of the original KV-cache footprint, state-of-the-art quality.
The Problem: KV-Cache Memory Explodes at Long Context
In a standard Transformer, every generated token has to attend to every previous token. To avoid recomputing, the model caches the Key and Value vectors for every past token — the KV-cache. That cache grows linearly with sequence length and linearly with the number of attention heads.
At 1 million tokens — DeepSeek V4's native context window — a vanilla KV-cache becomes the single biggest memory bottleneck on inference hardware. Earlier DeepSeek work (V2, V3) tackled this with Multi-head Latent Attention (MLA), compressing across the head dimension. V4 takes a fundamentally different route: compress along the sequence dimension itself, where the real growth happens.
Compressed Sparse Attention (CSA)
CSA is the fine-grained compressor in DeepSeek V4. Instead of storing one KV entry per token, it groups small windows of tokens — typically four — into a single compressed Key-Value pair.
Token-level compressor
Data-dependent weighting
Overlapping windows
The net effect: CSA preserves the local resolution the model needs for precise next-token prediction, while still cutting the cache to a fraction of the original.
Heavily Compressed Attention (HCA)
For the truly long stretches of context, DeepSeek V4 falls back to HCA — a much more aggressive compressor that bundles up to 128 tokens into a single KV entry.
HCA acts as a global summary. Individual fine-grained details are sacrificed, but the model can suddenly reason over enormous spans of text without compute or memory blowing up. In practice, HCA layers behave like a "long-range index" that the precise CSA and full attention layers can refer back to.
Low-Rank Queries & Grouped Output Projection
Compressing the KV-cache solves memory. To also cut compute, DeepSeek V4 shrinks the projection matrices themselves:
| Component | What gets shrunk | Approx. size vs. baseline |
|---|---|---|
| Low-rank queries | Query projection matrices factorised into two smaller matrices. | ~15.8% |
| Grouped output projection | Output projection shared across grouped heads. | ~14.3% |
| KV-cache (CSA + HCA) | Per-group compressed entries instead of per-token. | ~2% |
Figures based on the architecture summary in Jia-Bin Huang's DeepSeek V4 explainer.
The Hybrid Layer Stack
The real cleverness of Compressed Attention is where each variant gets used. DeepSeek V4 doesn't apply one technique uniformly — it stacks them by depth:
Early layers — HCA
Cheap global summary of the entire context window.
Middle layers — alternating HCA and CSA
Balance between long-range overview and local detail.
Final layer — full uncompressed attention
Maximum precision for the output token, no information loss.
Because the final layer is uncompressed, the model still gets a clean, high-resolution view when it actually generates the next token. The expensive compute is concentrated where it matters most.
Why Compressed Attention Matters
- Million-token context becomes practical. Memory is no longer the wall — hardware that previously couldn't host long-context inference now can.
- Inference cost collapses. Smaller KV-cache + smaller projections means cheaper serving, which directly enables DeepSeek V4 Flash's $0.14 / 1M input pricing.
- Open-source impact. With MIT-licensed weights, the entire community can inspect and reuse the Compressed Attention design.
- Architecture trend. Sequence-dimension compression is likely to become standard in the next wave of long-context models.
Frequently Asked Questions
Try DeepSeek V4 in your browser
Access DeepSeek V4 Pro and Flash directly from any webpage with the AI Sidebar Chrome Extension.
Install AI Sidebar