What Actually Fits and How Fast
| Board | RAM | Models (Q4_K_M) | Typical speed |
|---|---|---|---|
| Pi 5 | 8GB | 0.6B – 4B | 1–2B: 10–15 t/s · 3–4B: 5–8 t/s |
| Pi 5 | 4GB | 0.6B – 2B | 5–10 t/s |
| Pi 4 | 4–8GB | 0.6B – 1.5B | 3–6 t/s — patience required |
| Pi Zero 2 | 0.5GB | None practically | — use it for clients, not models |
The RAM math: a Q4_K_M 4B model is ~2.5GB, a 3B ~2GB, a 1.7B ~1GB — plus the OS and KV cache. That's why 8GB is the real Pi AI spec. The RAM-ranked guide has the full size table.
Setup: Ollama on Pi OS (10 Minutes)
# 1. Install Ollama (official Pi OS / ARM64 support)
curl -fsSL https://ollama.com/install.sh | sh
# 2. Pull a model that fits — 3B is the sweet spot
ollama pull smollm3:3b # fast
ollama pull qwen3.5:4b # quality (8GB board)
ollama pull qwen3:1.7b # safe on 4GB
# 3. Chat
ollama run smollm3:3b "Explain what a vector database is."
# 4. Serve for other devices on your network
ollama serve
# Other machines: OLLAMA_HOST=0.0.0.0:11434 python client.pyllama.cpp alternative (more control, slightly faster): build with make -j4 and run llama-cli -m model-q4_k_m.gguf -p "..." — the GGUF run guide covers it. Enable the right flags in your build; the NEON optimizations matter a lot on ARM.
Realistic Pi AI Projects (2026)
🗣️ Private voice assistant
Whisper (STT) + a 1–3B model + Piper/Kokoro (TTS) — a fully local, always-on voice box for ~15W. The full stack is in local voice assistants.
📚 Offline RAG kiosk
A Pi serving Q&A over a few hundred documents via ChromaDB + a small embedding model. Chunk carefully and keep the corpus small — the 8GB RAG guide is the template.
🏠 Home-automation brain
Natural-language control of Home Assistant: the model turns "dim the lights after 9pm" into structured tool calls. Small models handle this well.
📝 Sensor-log summarizer
A nightly job that summarizes the day's logs or sensor data into a readable report — batch work where 5 t/s doesn't matter.
The Honest Limits
- No large models. 7B+ on a Pi means extreme quantization and 1–2 t/s — technically possible, practically painful. The sub-12B rankings assume real computers; on a Pi, stay ≤4B.
- Long context is slow. The KV cache grows in CPU RAM and every token re-reads the model — keep contexts short (KV cache quantization helps a little).
- The Hailo AI kit is for vision, not text. The Pi 5's Hailo-8 accelerator is excellent for object detection and pose — on-device vision patterns — but LLM support is limited; CPU is still your text path.
- SD card I/O matters. Use a good A2 card or NVMe hat — swapping models off a slow card makes everything worse.
Frequently Asked Questions (FAQ)
Can a Raspberry Pi run an LLM?
Yes — small models (0.5B to 3B at Q4 quantization) run on a Pi 5 at usable speeds. Expect roughly 5–15 tokens per second on a Pi 5 with 8GB RAM. Larger models are possible with heavy quantization but become impractical.
What is the best LLM for a Raspberry Pi?
In 2026 the practical picks are Qwen3.5-4B (best quality per GB on Pi 5 8GB), Gemma 4 E4B, SmolLM3-3B (very fast), Qwen3-1.7B and 0.6B (older Pi 4 / 4GB boards), and LFM2.5-1.2B-Thinking for on-device reasoning. Quantize to Q4_K_M and test the actual speed before committing.
How fast is an LLM on a Raspberry Pi 5?
Realistic figures: 1–2B models run ~10–15 t/s, 3–4B models run ~5–8 t/s on a Pi 5 with 8GB RAM and a good (AVX/NEON-optimized) build. That's usable for chat and simple automation, slow for long documents.
Should I use Ollama or llama.cpp on a Pi?
Ollama is the easiest (one install, model pulls) and works well on Pi OS. llama.cpp gives more control (custom builds, KV cache flags) and slightly better performance. For most Pi projects, start with Ollama; switch if you need the knobs.
What are realistic Raspberry Pi AI projects?
A private voice assistant (Whisper + small LLM + TTS), a local RAG box for a few hundred documents, home-automation natural language, offline chat kiosk, and sensor-log summarization. Pi AI is for small, focused, always-on jobs — not general assistants.
Does the Pi 5's AI kit (Hailo) help with LLMs?
The Hailo-8 AI kit accelerates vision models and some small transformer inference, but LLM support is limited — it shines for object detection, pose, and similar vision workloads. For text LLMs, the CPU with an optimized llama.cpp build is still the main path.