DeepSeek V4.1 Flash grows as its KV cache shrinks 4x

The “one-quarter memory” claim for DeepSeek V4.1 Flash is not one number but three different numbers attached to three different places. The global KV cache that always sits in GPU memory (HBM) is 890 bytes per token, about a quarter of V4 Flash. The persistent cache left on SSD or host memory is about one-eighth, and against the first generation, DeepSeek-V1, it is about 1/437. The 890 bytes is not a value split across 40 layers but the sum of only the four layers that create new KV, and that sum shrank even as the parameter count nearly doubled from 284B to 552B. DeepSeek released the model on September 10 and described the design in a technical report on September 17 (arXiv 2609.19969, titled “Pushing the Limits of KV Cache Compression”). All dates in this article are in UTC.

What changed since then is on the model and engine side. On October 17 the model was replaced with a 1017 version, and the KV-related values in the 1017 repository’s config file (40 layers, the layer numbers that create KV, the indexer candidate size, the recent window of 128) all match the September checkpoint. So the numbers below also hold for the 1017 you download today. vLLM has supported the model since launch day, published how it raised throughput sharply in an October 7 blog post, and its recipe updated on October 25 recommends 1017 as the checkpoint. This article takes the numbers one at a time and reads what each one counts and what it is compared against.

Six numbers and their denominators

The same “it shrank” tells a different story when the denominator differs. The table below splits six lines of numbers from the report and the model card by what they count and what they are compared to.

Number	What it counts	Compared with
890 bytes	Global KV one token leaves in HBM (main KV and indexer key)	Absolute value, no comparison
About 4x smaller, about 437x smaller	Per-token size of the same global KV	V4 Flash, DeepSeek-V1 respectively
About 1/8	Persistent cache left on SSD or host memory for the same workload	V4 Flash
8B, 16B	Parameters active for one token (prefill, generation)	Out of 552B total
72+ hours, minutes	How long global KV and SWA KV stay in the cache	Each other
256x context, 1/4 more	Decode compute for one token when context grows from 4K to 1M tokens	The value at 4K tokens

All six lines leave out the same things. The recent 128-token window (SWA) cache attached to every layer, the model weights and intermediate compute values are not in the 890 bytes. The later part of this article brings in these outside numbers as well.

How tall it stacks up to 1M context

First, how much the 890 bytes comes to as context length grows. The chart below multiplies the per-token size of three generations by the token count and converts to GB. The vertical axis is logarithmic, so each gridline is 10x. The dotted horizontal line is the 141GB of memory on one H200.

Figure 1. Total global KV by context length. The V4 Flash and V1 lines are our own calculation, multiplying 890 by the “about 4x” and “about 437x” stated in the report; V1 does not actually support these lengths, so its line is a conversion

The V4.1 Flash line reaches 0.89GB at 1M tokens. At the same length V4 Flash is about 3.6GB, and if the V1 structure were stretched as is, it would be about 389GB, already past a single H200 at about 360K tokens. It is also worth noting that the three lines are parallel. Double the context and the cache honestly doubles, so what the structure changed is not the slope but the starting height. The point of this chart is that one 1M-token session stays under 1GB, which is why stacking dozens of sessions on one server becomes arithmetically possible for the first time.

Four of 40 layers spend the bytes

The 890 bytes is not spread evenly over 40 layers. Figure 2 lays the 40 layers out one cell each and colors only the layers that write new KV in orange.

Figure 2. Roles of the 40 layers (layer numbers start at 0). Layer numbers are from the public config file; the bytes are our calculation

The orange cells are encoder layers 2, 8 and 14 and decoder layer 20, four in all. Each kind of cell does a different job.

• Full: creates its own main KV and indexer key and also picks the 512 positions to read itself.
• Reindex: uses the main KV and indexer key of an earlier layer as is, but re-picks the positions to read with its own query. These are decoder layers 24, 28, 32 and 36.
• Reuse: takes both the KV and the positions to read from an earlier layer. It builds no indexer query and scores nothing, so by the report it runs with 15 prefill kernels and 11 decode kernels. There are 30 such layers.
• SWA-only: the first two layers (0 and 1) look only at the recent 128 tokens, with no global attention.

How Reindex picks where to read also has two stages. The first Full layer of the decoder (layer 20) scans every visible position and builds a candidate pool of 16,384 positions (2,048 blocks with 8 positions each). The later Reindex layers then score only inside that pool and pick the top 512. Even at 1M tokens of context, the search range of the later layers is fixed at 16,384, and the expensive full scan happens only once.

The math that makes 890 work, and 10,324 if everything were new

The report gives only the total of 890 and prints no per-item table. But adding it up yourself from the published storage format gives exactly 890. The main KV is stored in FP4 (E2M1, 0.5 bytes per value) with an 8-bit scale attached every 16 channels, and the indexer key uses MXFP4. The size of one entry is as follows (our calculation).

• One main KV entry: 512 values × 0.5 bytes + 32 bytes of scale = 288 bytes
• One indexer key entry: 128 values × 0.5 bytes + 4 bytes of scale = 68 bytes
• The encoder packs 2 tokens into one KV entry, so it is half per token (144 + 34 = 178); the decoder does not pack, so 356

The three encoder Full layers give 178 × 3 = 534, and the one decoder Full layer gives 356, for a sum of 890. Computing the same format for “if all 38 CSA layers were Full” gives 18 encoder layers × 178 = 3,204 plus 20 decoder layers × 356 = 7,120, or 10,324 bytes, so layer-to-layer reuse alone makes an 11.6x difference.

Figure 3. 890 versus 10,324 on the same scale (top two rows), and the actual 890 magnified 11.6x to show its composition (bottom row). The 10,324 bar is our assumed value for every layer being Full

In the actual values the encoder main KV (432) takes nearly half, and both indexer keys together are 170 bytes. That means the device that picks where to read is itself cheap. Still, 10,324 is an assumption about a different structure from V4 Flash’s actual size (about 3,560), so do not put the two numbers on the same scale and read a winner from them.

There is also a number for the precision lowered to 4 bits. FP4 has a narrow representable range, but the report explains that because a scale is attached every 16 channels, the maximum representable value reaches 2,688 (448 × 6), while KV values after rotary position encoding (RoPE) stay far below that, within about 22.6. Quantization is applied after RoPE and was accounted for from training. As a result, storage is roughly half that of V4’s FP8 main KV. The SWA KV is kept in FP8 without compression and has a fixed size regardless of context length, so it is not in the 890.

Prefill stops at the encoder

Where the 890 bytes gets made becomes visible when you follow the path through the model. In the figure below, the left box is where prefill, which reads a long prompt, passes through, and the right is where generation adds more.

Figure 4. Layers that prefill and generation pass through. Active parameters of 8B and 16B are model card values; retention times are report values

A prompt only passes the 20 encoder layers to finish prefill, so about 8 billion parameters are active. The decoder’s global KV is not made by the decoder layers from their own state; layer 20 projects it once from the encoder’s last hidden state, and the remaining decoder layers keep using it. The generation step goes through both encoder and decoder, so about 16 billion are active. The report explains that prefill complexity, which was N·L for L layers and N tokens, drops to N·L/2 plus the recent-window part (n_win·L/2), roughly halving the computation. In the scene where an agent re-reads a long history every time it calls a tool, that half is directly waiting time.

The figure that per-token compute barely grows as context grows needs to be read with its conditions. Figure 2 of the report shows that when context grows 256x, from 4K to 1M tokens, the decode compute for one token grows by only 1/4, but that compute is counted weighted by precision (BF16 ×1, FP8 ×0.5, FP4 ×0.25). The more work is done in FP4, the smaller the same work scores, so this is a comparison of the structure’s compute, not the chip’s actual elapsed time.

The roots of this idea go back to research from May 2024. On May 7, DeepSeek-V2’s MLA shrank KV into a latent vector (a 93.3% reduction against the 67B model, per the paper’s abstract), on May 8 YOCO made the global KV once in the lower decoder for the upper layers to use, and on May 21 CLA shared key and value heads between adjacent layers. V4.1 inherits the cross-layer sharing side and stacks a sparse indexer, FP4 storage and recent-window restoration in one model.

The one-eighth came from a different place than the one-quarter

The one-eighth of the persistent cache is not the same story as the one-quarter of the 890 bytes. V4 kept the full prefix of the global KV and, separately, SWA KV at the end of the prompt and the end of the output, and according to the report the SWA KV took nearly half of the persistent cache capacity. V4.1 takes the SWA KV out of the persistent cache, which removes half, and the remaining global KV shrinks to a quarter, which together make one-eighth.

Figure 5. How a persistent cache of 100 becomes 12.5 (one-eighth). The first step is our calculation setting the report’s “nearly half” to 50, and the second step is one-quarter of the remaining 50

The two caches also have different lifetimes. The global KV is guaranteed at least 72 hours in the persistent cache so it can be reused when the same long prefix comes back, while the SWA KV sits only for minutes in a distributed pool built from 10% of each machine’s host memory (DRAM). When the same prefix returns after the SWA state has vanished from the pool, a SWA boundary replay re-reads the last 128 tokens of the prefix to restore the encoder-side state. An exact restoration would require re-running as many tokens as the layer count L times 128, but this is reduced to an approximation that reads only the last 128 tokens. On the decoder side, each prefill passes the last 128 tokens of the prompt through the decoder layers using the encoder output, uses them only for the first decode step, and does not store this state in the prefix cache.

So the trade is saving storage at the cost of a little more computation. The report says it saw no systematic performance degradation in the settings it evaluated, and vLLM also implemented this replay and reported no meaningful difference on GSM8K and GPQA (within about 1.5 standard error).

Recounting the 890 from the config file

Without downloading the weights, a single 3KB config file lets you confirm the layer layout above and the 890 yourself. The script below reads config.json from the 1017 repository, separates each layer’s role by its compression ratio and two lists of layer numbers, and adds up the bytes of the Full layers. The 288 and 68 are the entry sizes computed in the previous section.

curl -sL https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/config.json -o config.json
python3 - <<'PY'
import json
c = json.load(open("config.json"))["text_config"]
kv, ix = set(c["kv_source_layer_ids"]), set(c["index_source_layer_ids"])
cr = c["compress_ratios"]
kinds, total = {}, 0
for i in range(c["num_hidden_layers"]):
    k = "swa-only" if cr[i] == 0 else "full" if i in kv else "reindex" if i in ix else "reuse"
    kinds.setdefault(k, []).append(i)
    if k == "full":
        total += (512 * 0.5 + 32 + 128 * 0.5 + 4) / cr[i]
print("compress_ratios entries:", len(cr))
for k, v in kinds.items():
    print(f"{k:9s}{len(v):3d}", v if len(v) < 9 else v[:3] + ["..."])
print("global KV bytes per token:", total)
PY

compress_ratios entries: 43
swa-only   2 [0, 1]
full       4 [2, 8, 14, 20]
reuse     30 [3, 4, 5, '...']
reindex    4 [24, 28, 32, 36]
global KV bytes per token: 890.0

That kv_source_layer_ids is 2, 8, 14, 20 is the basis for “four layers create new KV,” and adding 24, 28, 32 and 36 from index_source_layer_ids makes the Reindex layers. In the compression-ratio list the first two layers are 0 (no global attention), the next 18 layers are 2, and the 20 decoder layers are 1, but the list is 43 long, so three more zeros trail at the end. The script read only the first 40 as layers, and the sum comes out as 890.0. candidate_topk_blocks 2,048, candidate_block_size 8 and index_topk 512 can be read from the same file, which is where the earlier 16,384 and 512 came from.

The numbers outside the 890

To use these numbers in a capacity calculation for one server, you need the numbers outside the 890. The 0.89GB of global KV for a 1M-token session is small partly because the cache is small, but also because the weights next to it are so large. The values the vLLM recipe lists are as follows.

Checkpoint size

About 511GB (476GiB). Of this, the Engram table is 183.1GiB and the routed experts are 259.5GiB

Minimum VRAM

614GB (including a 1.2 headroom factor)

8x H200 node

Fits in 1,128GB, and with TP4 and Engram offload, 81.2GiB of weights plus 38.5GiB of KV per GPU

Global KV at 1M tokens

0.89GB, about 0.17% of the checkpoint (our calculation)

Fixed share

SWA KV of the recent 128 tokens per layer (FP8, uncompressed), independent of context length

In the same place the recipe notes that what sets capacity is usually the weights and batch size, not the cache, so measure first whether your context and batch fit. With 511GB of weights and a cache of under 1GB, the gain from cutting the cache to a quarter comes from “more sessions” and “longer context,” not from “a smaller server.” The shorter the context, the larger the share by which the fixed part (SWA KV) outweighs the 890 side.

What vLLM’s 5.3x is measured against

The numbers the engine produced have yet another denominator from the model’s numbers. vLLM has supported V4.1 Flash since launch day (September 10), the version where this architecture first landed is 0.30.0, and the recipe updated on October 25 requires 0.31.0 or later. The throughput gains the October 7 blog advertises come from the following conditions.

Metric	Value	Conditions and baseline
Low-latency throughput	1.9x	SemiAnalysis AgentX, TP4 + FlashInfer, against vLLM’s launch-day implementation
High throughput	About 5.3x	Same AgentX, DEP2, 150 TPS constraint, same baseline
Time to first token (TTFT)	About 70% lower	Near 100K throughput, three optimizations combined
Decoder replay + CUDA graph	30-40% less prefill compute time	Single request, GB200, prefix cache off, median of 3 runs

The key point is that the baseline is not an earlier version of the model but vLLM’s own launch-day implementation. The 5.3x is not an improvement DeepSeek made in the model but an improvement from the engine being tuned to use this structure properly. The same blog also shows how the structure was implemented. Decoder layer 20 runs on every token, but layers 21 to 39 run only on the last 128 tokens of each request, and on a prefix cache hit the SWA boundary replay is on by default and can be turned off with --no-swa-bounded-replay. Without the CUDA graph, the replay is slower than the baseline on short prompts, by up to 12% at DEP2 and 1K tokens.

The storage format also differs by engine and hardware. The blog says KV compressed with NVFP4 is 45% smaller than the earlier FP8 KV cache, and the recipe’s Blackwell settings include both --kv-cache-dtype fp8 and the indexer’s mxfp4. The 890 bytes is counted in the structure DeepSeek defined, and whether the engine you run stores in the same format is something to check against the engine’s documentation and the actual GPU memory use. KV offloading was not needed across the blog’s benchmarks, and the blog only says it expects it to help with many concurrent users or with environments that split prefill and decode across servers.