How to Convert a Hugging Face Model to GGUF (Step by Step)

Q: How much disk space do I need to convert a model?

Roughly 2-3x the model's FP16 size: the original safetensors weights, the intermediate FP16 GGUF, and the final quantized GGUF. For a 7B model that's about 14GB for the original, 14GB for the FP16 GGUF, and ~4GB for the Q4_K_M result — plan for ~30GB free.

What You Need

A Hugging Face model repo containing: the weights (.safetensors or pytorch_model.bin), config.json, and tokenizer files (tokenizer.json and/or tokenizer.model)
Python 3.8+ with the huggingface_hub library installed (pip install huggingface_hub)
The llama.cpp repository — cloned locally for its Python converter, and compiled for llama-quantize
Disk space: roughly 2–3× the model's FP16 size (a 7B model → plan for ~30GB free)

💡 First, check if you even need to convert. Hugging Face already hosts GGUF versions of most popular models. If you see .gguf files on the model page (or on a directory like Local AI Zone), download those instead — converting is for models that don't have a GGUF yet, or your own fine-tunes. Our GGUF explainer covers the format itself if you need a refresher.

Step 1: Download the Model from Hugging Face

Download the full snapshot (weights + config + tokenizer) into a local folder. The current Hugging Face CLI is hf (the legacy huggingface-cli download still works but is deprecated):

hf download <repo_id> --local-dir ./model

Or from Python:

from huggingface_hub import snapshot_download
snapshot_download(repo_id="<repo_id>", local_dir="./model")

Example with a real model:

hf download Qwen/Qwen2.5-7B-Instruct --local-dir ./qwen2.5-7b

Step 2: Get llama.cpp

Clone the repository and install the Python dependencies for the converter:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
pip install -r requirements.txt

Then build the C++ tools (this gives you llama-quantize, llama-cli, and friends):

cmake -B build
cmake --build build --config Release -j

On Windows use cmake --build build --config Release -j -- /p:PreferredToolArchitecture=x64 if you hit build issues; the tools land in build/bin/.

Step 3: Convert to Full-Precision GGUF

Run the converter on the model folder you downloaded:

python convert_hf_to_gguf.py ../qwen2.5-7b \
    --outfile ../qwen2.5-7b-f16.gguf \
    --outtype f16

What's happening: the script reads the safetensors weights, config.json, and tokenizer, and writes one self-contained GGUF file at full precision (f16 or bf16 — pick bf16 if the model card specifies it). For a 7B model this produces roughly a 14GB file.

⚠️ Architecture support: the converter only handles decoder-only architectures in llama.cpp's supported list — Llama, Mistral, Gemma, Phi, Qwen, DeepSeek, and many more. Encoder-decoder or unsupported architectures will fail with a clear error. Most open-weight chat models convert without trouble.

Step 4: Quantize It

The FP16 GGUF runs, but it's huge. Quantization shrinks it to fit your RAM/VRAM:

./build/bin/llama-quantize ../qwen2.5-7b-f16.gguf \
    ../qwen2.5-7b-q4_k_m.gguf Q4_K_M

Swap Q4_K_M for any supported level: Q5_K_M, Q6_K, Q8_0, and so on. Want more than one? Quantize the same FP16 file into several sizes — that's exactly how model hubs publish a whole family from one conversion. See Q4_K_M vs Q8_0 to pick the right level.

Optional: split a large GGUF into multiple files for FAT32 or USB drives with llama-gguf-split:

./build/bin/llama-gguf-split --split-max-size 4G \
    ../model-q4_k_m.gguf ../model-q4_k_m-split

Step 5: Run It

Your quantized GGUF now works with every GGUF-compatible tool:

llama.cpp CLI: ./build/bin/llama-cli -m ../qwen2.5-7b-q4_k_m.gguf -p "Hello"
llama.cpp server (OpenAI-compatible API): ./build/bin/llama-server -m ../qwen2.5-7b-q4_k_m.gguf --port 8080
Ollama: point a Modelfile at the GGUF — FROM ./qwen2.5-7b-q4_k_m.gguf, then ollama create qwen2.5-7b -f Modelfile
GGUF Loader: drag the .gguf file into the desktop app and chat — ggufloader.github.io

New to running GGUF models? Our beginner's guide covers install, model picks, and troubleshooting, and the 2026 tool roundup helps you pick a runtime.

Faster Alternatives (Skip the Local Setup)

Option	When to use it	Notes
GGUF-my-repo (Hugging Face Space)	One-off conversion, no local install	Free online Space: paste a model ID, get GGUF files back
Ollama auto-convert	You have a local HF model and want it in Ollama	`ollama create` from a `FROM /path/to/model` Modelfile converts automatically
Download existing GGUF	Almost always	If the model already has GGUF files (or a curated directory like Local AI Zone lists them), download — converting is unnecessary

Troubleshooting

"Unknown model architecture": the converter doesn't support this architecture yet. Check llama.cpp's issue tracker for a converter PR, or use a different model.
Missing tokenizer: the repo needs tokenizer.json or tokenizer.model. Some repos store it with git-lfs — make sure the snapshot download included it.
Out of disk space: convert in a directory with room for original + FP16 + quantized copies, then delete the intermediates.
Wrong output type: match --outtype to the model card (f16 vs bf16); mismatches can cause slow or broken inference on some backends.
Garbage output after quantizing: try a higher quant (Q6_K/Q8_0) of the same model, or re-run from the FP16 file — never quantize an already-quantized file.

Frequently Asked Questions (FAQ)

What files do I need from Hugging Face to convert a model to GGUF?

The model weights (safetensors or pytorch_model.bin), a config.json describing the architecture, and the tokenizer files (tokenizer.json and/or tokenizer.model). A snapshot download pulls all of these automatically.

Can I convert any Hugging Face model to GGUF?

Only decoder-only LLM architectures supported by llama.cpp — Llama, Mistral, Gemma, Phi, Qwen, DeepSeek, and many more. The converter needs the model's architecture to be in llama.cpp's supported list, which covers the vast majority of open-weight chat models.

Do I need a GPU to convert a model to GGUF?

No. Conversion and quantization run on CPU. You need Python for convert_hf_to_gguf.py and a compiled llama-quantize binary, but no GPU is required — quantization is just number crunching over the weight files.

What is the difference between converting and quantizing?

Converting turns the original safetensors/pytorch weights into a single GGUF file at full precision (usually f16 or bf16). Quantizing then shrinks that GGUF file to a lower precision like Q4_K_M to fit in less memory. Both use llama.cpp tools.

Is there a way to convert without installing llama.cpp locally?

Yes. The GGUF-my-repo Space on Hugging Face converts a model online. Ollama also auto-converts a local Hugging Face model when you create it from a Modelfile. And many models already ship as GGUF on Hugging Face — downloading an existing GGUF is almost always easier than converting.

How much disk space do I need to convert a model?

Roughly 2–3× the model's FP16 size: the original safetensors weights, the intermediate FP16 GGUF, and the final quantized GGUF. For a 7B model that's about 14GB for the original, 14GB for the FP16 GGUF, and ~4GB for the Q4_K_M result — plan for ~30GB free.