The Quantization Ladder, Decoded
| File | Bits/weight | Size for 7B | Quality | Use case |
|---|---|---|---|---|
| F16 | 16 | ~13GB | Full | Source for conversions; exactness |
| Q8_0 | 8 | ~7.2GB | Near-lossless | Quality-critical local runs — see the comparison |
| Q6_K | 6 | ~5.5GB | Very good | Spare-memory quality bump |
| Q5_K_M | 5 | ~4.8GB | Good | Quality + size balance |
| Q4_K_M ⭐ | 4 (mixed) | ~4.4GB | Good (default) | The standard pick — best quality-per-GB |
| Q3_K_M | 3 | ~3.5GB | Noticeable loss | Small GPUs, CPU laptops |
| Q2_K | 2 | ~2.8GB | Significant loss | Last-resort memory fits |
Reading the Names: Q4_K_M, Q8_0, F16
- Q = quantized. The number is the bit level (Q4 = 4-bit average, Q8 = 8-bit).
- _K = K-quant scheme. A smart mixture: most layers at the target bits, a few sensitive layers at higher precision. _S (small) and _L (large) vary the mix; _M (medium) is the balanced default.
- _0 = uniform scheme. All weights at the same bits (Q8_0 is the common one) — simpler, predictable.
- F16 = half float, unquantized — the full model, used as the conversion source.
Why the mixture matters: not all weights are equal. Attention and output layers are more sensitive to precision than feedforward layers, so K-quants protect the sensitive ones — which is why Q4_K_M routinely outperforms a naive uniform 4-bit at the same size.
The Size Math (and Why It's Fuzzy)
Naive estimate: parameters × bits ÷ 8
7B at 4 bits → 7,000,000,000 × 4 / 8 ≈ 3.5GB
Reality for Q4_K_M: ≈ 4.4GB (+26%)
Reasons: mixed-precision layers (K-quant keeps some
tensors at 6/8 bits), tokenizer + metadata, and
attention tensors stored at higher precision.So the rule of thumb: compute the naive number, then add ~30–60% for K-quant files. The exact size is on the repo page — the math is for planning. The RAM-ranked guide does this planning for you across hardware tiers.
How to Pick: The 30-Second Decision
| Your hardware | Pick |
|---|---|
| 8GB VRAM / 16GB RAM laptop | Q4_K_M (Q3_K_M if it must co-run apps) |
| 12–16GB VRAM | Q4_K_M → Q5_K_M / Q6_K if memory allows |
| 24GB+ VRAM / 32GB+ unified | Q6_K or Q8_0 — quality now matters more than size |
| CPU-only, slow RAM | Q4_K_M (or Q3_K_M for speed) — see the speed guide |
| Converting your own model | Start from F16, convert to target quant |
🚀 The rule that never changes
Start at Q4_K_M. It's the best quality-per-GB in the ladder, which is why it's the community default everywhere — including this blog's RAM-ranked download tables and the conversion tutorial. Only move up (Q5/Q6/Q8) with spare memory, or down (Q3/Q2) when hardware forces it.
Frequently Asked Questions (FAQ)
Why does one model have so many GGUF files?
Each file is the same model at a different quantization level — how many bits each weight is stored in. More bits = bigger file = higher quality; fewer bits = smaller file = faster and lighter. Model repos publish the full ladder so you can pick the exact size/quality trade-off for your hardware.
How do I calculate a GGUF file's size?
Roughly: parameters × bits-per-weight ÷ 8. A 7B model at 4 bits ≈ 7e9 × 4 / 8 ≈ 3.5GB. Add ~5–10% for quantization metadata and tensors that stay at higher precision. The Q4_K_M file for a 7B is typically ~4.4GB in practice.
What do the GGUF names mean (Q4_K_M, Q8_0, F16)?
The prefix is the bit level (Q4 = 4-bit, Q8 = 8-bit, F16 = half float). The suffix is the scheme: _K_M means the K-quant "medium" mixture (most layers at 4-bit, a few at higher), _0 is a uniform scheme, and _S/_L are small/large K-quant variants. Q4_K_M is the popular default because it packs quality into size.
Which GGUF file should I download?
Start with Q4_K_M: the best quality-per-GB in the ladder. Choose Q5_K_M or Q6_K if you have spare memory and want more quality; Q8_0 for near-lossless; Q3_K_M or Q2_K only when hardware forces it. The RAM-ranked guide in this blog maps each quant to hardware tiers.
Why is the Q4 file bigger than parameters × 4 bits suggests?
Because not everything is 4-bit: K-quant schemes keep some weights and tensors at higher precision (the "K" mixture), plus the file stores tokenizer data, metadata, and attention tensors at 6/8 bits. The result is a Q4_K_M file running ~30–60% above the naive 4-bit estimate.
Can I change a model's quantization after downloading?
Yes — llama.cpp's quantize tool converts GGUF between levels offline: ./llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M. Start from F16 or Q8_0 for best results; re-quantizing an already-quantized file compounds quality loss. The conversion guide covers the full workflow.