KV Cache Quantization: What It Is and Why It Matters (2026 Guide)

Published: August 8, 2026 — When a local model "won't fit," the culprit is often not the weights at all. It's the KV cache — the attention state that grows with every token of context. On a 128K-context run, the KV cache can dwarf the model itself. KV cache quantization compresses that state, letting you run long-context workloads on hardware that would otherwise run out of memory, with quality loss that ranges from zero to small depending on how far you push it.

⚡ Quick Takeaways

What Is the KV Cache?

Every transformer computes, for each token, a Key and Value vector used by attention. Rather than recomputing them on every step, inference engines cache them — that's the KV cache. It's a brilliant optimization, with one catch: it grows linearly with context length. Each new token adds new K/V entries for every layer and every attention head.

That's why a "7B model" can need far more than its weight size. On a 128K-token context with FP16 KV, a 7B model's cache is roughly the same size as the model itself; for 1M-context models it's many times larger. The math:

KV bytes ≈ 2 (K and V) × layers × heads × head_dim × context_tokens × bytes_per_value

# Example: 7B model, 32 layers, 128 heads × 64 dim, 32K context, FP16
≈ 2 × 32 × 8192 × 32768 × 2 bytes ≈ 34 GB  (!!)

The exact per-model numbers vary, but the shape is universal: long context = KV cache dominates memory.

What Quantizing the Cache Buys You

KV precision Cache size (vs FP16) Quality impact Best for
FP16 / FP32 1x (baseline) None Max precision, memory to spare
Q8_0 ~0.5x Effectively none in most tasks Default — long context, any task
Q4_0 / Q4_1 ~0.25x Small — long-context recall, details Chat, summarization, memory-constrained
Q2/Q3 class ~0.15x Noticeable Experiments only

Practical effect: on an 8GB GPU, FP16 KV might cap you at ~8–10K context for a 7B model — but with Q8_0 KV you comfortably double or triple that, and Q4_0 pushes further still. For 8GB-RAM RAG setups this is often the difference between running and not running long documents.

How to Enable It in llama.cpp and Ollama

# llama.cpp (current syntax)
llama-server -m model-q4_k_m.gguf \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  -c 32768

# Older syntax, still accepted
llama-server -m model-q4_k_m.gguf -ctk q8_0 -ctv q8_0 -c 32768

# Most aggressive, for chat workloads
llama-server -m model-q4_k_m.gguf --cache-type q4_0 -c 32768

Ollama: recent versions expose KV cache settings through the modelfile — PARAMETER num_ctx controls the window, and server flags/`OLLAMA_KV_CACHE_TYPE` (where available) set precision. In practice most Ollama users never touch it: the server auto-selects a reasonable cache type for the model. If you're pushing context limits, switching to llama.cpp for full control is the move.

vLLM: --kv-cache-dtype fp8 (and fp8_e4m3 on newer GPUs) does the same thing server-side, with --max-model-len controlling the budget.

When to Use Which Setting

Workload Recommended KV
Legal/medical document analysis (precision) FP16 if it fits, else Q8_0
General RAG over long documents Q8_0 — the safe default
Chat, summarization, creative writing Q8_0, or Q4_0 on tight hardware
Code generation (long files) Q8_0 — syntax tokens are repetitive and compress well
8GB GPU / 16GB RAM laptop, big context Q4_0 KV + Q4_K_M weights — see the RAM guide

💡 Real-world note from building privacy-first assistants: the KV cache is also a privacy surface. On shared or rented hardware, cache eviction and swap can leave token data in memory longer than expected. For regulated workloads, prefer Q8_0 (smaller footprint, less swap) over letting the cache spill — and keep everything on your own machine, as Lawyer Assistant does.

Frequently Asked Questions (FAQ)

What is the KV cache?

The KV cache stores the Key and Value tensors computed for every token in the context so attention does not recompute them. It grows linearly with context length and is often the largest memory consumer on long-context workloads — easily matching or exceeding the model weights themselves.

Does KV cache quantization hurt quality?

Q8_0 KV cache costs essentially nothing measurable in most tasks. Q4_0 or Q4_1 saves more memory but shows small quality drops on long-context recall and detail-heavy tasks — acceptable for chat and summarization, risky for legal or medical precision work.

How much memory does the KV cache use?

A rough rule: about 1GB per 10K tokens for a 7B model at FP16 KV. Halving the precision (Q8_0) halves the cache; Q4_0 quarters it. For a 128K-context run the cache can exceed the model weights, which is exactly why quantization matters.

How do I enable KV cache quantization in llama.cpp?

Pass --cache-type-k q8_0 --cache-type-v q8_0 to llama-server or llama-cli, or use the newer unified --cache-type q8_0. Recent builds default to sensible per-layer settings; you can also set -ctk/-ctv in older syntax.

Is KV cache quantization the same as quantizing the model?

No. Model quantization (Q4_K_M etc.) compresses the weights — a one-time, fixed-size saving. KV cache quantization compresses the attention state, which grows with context. They stack: quantize weights for model size, quantize KV for long context.

When should I keep FP16 KV cache?

When you have the memory headroom and the task is precision-sensitive: long legal or medical documents, exact-number recall, or benchmark comparisons where you want zero variables. For everything else, Q8_0 KV is the sensible default.

Sources & Further Reading