Qwen's 128K Full-Attention Cache: 32 GiB in One Model, 8 GiB in Another
At 128K tokens, Qwen3-32B's calculated 16-bit full-attention KV payload is 32 GiB; Qwen3.8-27B's is 8 GiB. Weights and other state are extra.
Twenty-four GiB separates the full-attention caches of two Qwen models processing 128K tokens. Qwen3-32B needs 32 GiB and Qwen3.8-27B needs 8 GiB when both retain 131,072 tokens in one sequence at 16-bit precision, a difference calculated from their published configurations that can change a local AI hardware budget before anyone adds a faster GPU to the shopping cart. Both models still need their weights and other working memory.
Consider Qwen's Qwen3-32B Q4_K_M download, listed at 19.8 GB. Comparing that file with a GPU's advertised capacity leaves out the conversation. At your chosen context length, the cache can push total memory beyond the available capacity even when the weights fit.
The conversation has its own storage
During generation, an attention model saves intermediate representations called keys and values so that it can reuse earlier work; this KV cache holds numerical representations of the tokens being processed, including both the input and generated output. The cache has its own precision setting. A four-bit weight file does not automatically select a four-bit cache.
For one sequence with a conventional full-attention cache, the payload calculation is straightforward:
KV bytes = tokens × full-attention layers × KV heads × head dimension × 2 × bytes per value. The factor of two counts keys and values. FP16 and BF16 each use two bytes per value; GiB means 2³⁰ bytes.
Qwen3-32B's configuration specifies 64 attention layers, eight KV heads and a head dimension of 128. Multiplying 64 × 8 × 128 × 2 × 2 gives 262,144 bytes, or 256 KiB, for each cached token. Every additional 4,096 tokens therefore adds exactly 1 GiB of theoretical KV payload.
Converting the rounded 19.8 GB file listing gives approximately 18.4 GiB. Adding 32 GiB for the cache gives 50.4 GiB before buffers and other allocations, with the file size serving only as a proxy for resident weight storage.
Count the layers that actually keep a KV history
Qwen3.8-27B uses the hybrid architecture also used by Qwen3.5: its configuration lists 64 layers, arranged in groups of three Gated DeltaNet layers followed by one full-attention layer, so only 16 layers retain a conventional KV history, with four KV heads of 256 elements each.
That produces 16 × 4 × 256 × 2 × 2 = 65,536 bytes per cached token. The full-attention cache is one-quarter as large. Both models store 1,024 elements per key in each full-attention layer: eight heads of 128 versus four heads of 256. Count the layers.
Gated DeltaNet retains recurrent state, so the remaining 48 layers still consume memory; isolating the full-attention payload lets us compare the cache component that grows with each token, while leaving the additional recurrent-state allocation for a runtime-specific measurement rather than quietly treating it as zero.
| Total cached tokens | Qwen3-32B | Qwen3.8-27B |
|---|---|---|
| 32,768 | 8 GiB | 2 GiB |
| 65,536 | 16 GiB | 4 GiB |
| 131,072 | 32 GiB | 8 GiB |
| 262,144 | Beyond the cited supported limit | 16 GiB |
The calculation data records the formula, pinned source revisions and results.
Qwen's older model card specifies 32,768 native tokens and 131,072 with YaRN context extension. Qwen3.8's card specifies 262,144 natively. At those two context lengths, the newer model can represent twice the older model's extended token count with half its full-attention KV payload, but the usefulness of those tokens remains a separate question that a configuration file cannot answer.
The strongest counterargument: a smaller cache is not the whole job
A buyer needs a model that does the work. If the smaller-cache model makes more mistakes on the reader's actual coding or retrieval task, or requires a runtime that performs poorly on the available hardware, its memory advantage may buy little; selecting a model by this table alone would repeat the same mistake as selecting it by download size.
Software also changes what fits. Hugging Face documents dynamic caches, fixed-capacity caches, quantization and offloading, so a runtime can reserve unused cache capacity or keep some cache tensors in CPU memory; either choice changes the relationship between this calculated payload and the GPU allocation a monitoring tool reports.
Ollama documents a separate cache setting with f16 as the default, q8_0 using approximately half the memory and q4_0 approximately one-quarter, provided Flash Attention is enabled on a supported backend; these are approximate ratios because quantized storage carries overhead, and the documentation warns that quality changes depend on the model and task.
Those options can make the larger cache manageable. A useful comparison must therefore specify runtime, cache precision and workload, and a fourfold difference in one cache component cannot, by itself, predict a fourfold speedup or a fourfold reduction in total memory.
Limitations
This is a calculation of tensor payload from published configurations. We did not measure total GPU allocation, response speed, long-context accuracy or cache-quantization quality; the table excludes weights, execution buffers, allocator padding, vision processing, recurrent state and any extra state checkpoints a serving engine retains, so measure the actual runtime before treating these numbers as hardware requirements.
The calculation assumes one sequence without cache sharing or eviction. Parallel conversations can increase the bill, as Ollama's concurrency documentation explains, and two models can tokenize the same document differently, so equal token counts need not represent equal amounts of text.
What you can do before buying more memory
Write down the workload first: input tokens, room for generated output and simultaneous requests. Then check the attention layout, weight placement, cache precision and reported allocation in your runtime; Ollama's context documentation shows how to inspect configured context and processor placement with ollama ps, which makes a more useful starting point for troubleshooting than the number printed beside a download.
Start with the context your work needs, measure a representative long request, and test a supported eight-bit cache if memory is tight. Keep the outputs and compare their accuracy before accepting the saving. For these two models, that planning exercise exposes a 24 GiB full-attention cache difference at 128K tokens, a substantial hardware decision hiding inside a small configuration file.