How to Speed Up Local LLMs: CPU vs GPU vs NPU (2026 Guide)

CPU vs GPU vs NPU: What Each One Is Good At

	CPU	GPU	NPU
Best model size	Anything it can fit in RAM (slowly)	Up to VRAM capacity, fast	1–4B sweet spot
Speed (7B Q4)	3–10 t/s (DDR5) / 10–25 t/s (Apple Silicon unified)	30–100+ t/s	20–60 t/s on 1–3B
Power draw	15–150W	150–450W	2–15W
Bottleneck	Memory bandwidth	VRAM capacity + bandwidth	Toolchain + model support
Best for	Any model, anywhere, zero setup	Big models, agents, batch jobs	Phones, laptops, always-on assistants

The unifying rule: each generated token requires reading the entire model's weights from memory. Faster memory and smaller weights are the two things that make tokens faster. That is why quantization dominates every other optimization — it attacks the bottleneck directly. See Top 10 GGUF Models by RAM for size-matched picks.

The Speed Levers, Ranked by Impact

Quantize the model. Q4_K_M vs FP16 is a ~4–5x data-movement reduction. On CPU this is the difference between unusable and usable. (What is GGUF? explains the format.)
Match model size to memory bandwidth. A 3B model on DDR5 laptops is 2–3x faster than a 7B on the same machine — small models are the real CPU speedup.
Use the right build. llama.cpp with AVX2/AVX512, Metal (Apple), CUDA/Vulkan/ROCm offload — stock installs often run on the slowest path. Ollama and llama.cpp both handle this automatically in 2026, but verify your build log shows the accelerator.
Shrink the context. Long contexts grow the KV cache, which steals memory bandwidth from generation. See KV Cache Quantization for the fix.
Speculative decoding on GPU — a small draft model proposes, the big model verifies in parallel. 1.5–3x, output-identical. Details in our dedicated guide.
Batching. For APIs and agent loops, process multiple requests at once — throughput scales far better than latency. vLLM is the tool for this (Ollama vs vLLM).

Realistic Tokens-per-Second Targets (2026)

Hardware	Model (Q4_K_M)	Typical speed	Feels like
Phone NPU	1–3B	20–60 t/s	Fast, low power
Laptop CPU (DDR5)	3–8B	4–12 t/s	Usable chat
Apple Silicon (M-series)	7–14B	15–40 t/s	Fast — unified memory wins
Desktop CPU (DDR5)	7–14B	6–15 t/s	Usable, agent work OK
8GB GPU (RTX 4060-class)	7–8B	40–80 t/s	Very fast
16GB GPU (RTX 4070/4080-class)	12–14B	50–110 t/s	Cloud-feel
24GB+ GPU	30B–70B (offloaded)	15–40 t/s	Big-model territory

💡 If you're under 4 t/s, check three things in order: is the model quantized (Q4_K_M or lighter)? Is the build using the right instruction set / accelerator? Is the context window much longer than needed? Fixing those three fixes almost every slow setup.

The NPU Shift: 2026's Quiet Speedup

NPUs (neural processing units) shipped in virtually every new laptop and phone by 2026 — Qualcomm, AMD, Intel, Apple — and the toolchains finally caught up. Small models on NPUs now run 2–5x faster than the same model on the same machine's CPU, at a fraction of the power. The trade-off is coverage: NPU support is per-model and per-quant, drivers vary, and models above ~4B rarely fit the NPU's SRAM budget.

Where NPUs genuinely shine in 2026: on-device phone assistants, voice pipelines (local voice assistants), and always-on agentic loops on laptops — workloads where 2W of inference instead of 30W changes the product. The Liquid AI LFM family (covered in our LFM guide) was designed from day one with NPU deployment in mind.

Quick Fix Checklist

✅ Re-download the model at Q4_K_M instead of Q8 or FP16 (is Q8 worth it?)
✅ Switch to a smaller model — sub-12B rankings have per-hardware picks
✅ Confirm the runtime log shows GPU/NPU offload, not pure CPU
✅ Trim context window to what the task needs
✅ Turn on flash attention if supported (big context win)
✅ Enable speculative decoding on GPU builds
✅ For servers: switch to vLLM with batching

Frequently Asked Questions (FAQ)

What is a good tokens-per-second speed for a local LLM?

For interactive chat you want at least 8–15 tokens per second; 20–40 t/s feels fast; 50+ t/s approaches cloud-service feel. For agent loops and batch work, throughput matters more than latency, and 100+ t/s on smaller models is common.

Does quantization speed up local LLMs?

Yes — this is the single biggest speed lever. A Q4_K_M model is roughly 4–5x smaller than FP16, so memory bandwidth (the usual bottleneck on CPU) moves 4–5x more data per second, giving a near-proportional speedup with a small quality cost.

Can NPUs run LLMs faster than CPUs?

For small models (1–4B) on phones and laptops, yes — NPUs deliver 2–5x the tokens per second of a CPU at a fraction of the power. For models above ~8B, the GPU is still the workhorse; NPU drivers and toolchains are maturing fast in 2026.

Why is my local LLM so slow on CPU?

Token generation is memory-bandwidth-bound: every token requires reading the whole model from RAM. Slow RAM, an unquantized model, and a small batch are the usual culprits. Quantize, enable an AVX2/AVX512 build, and use a smaller model — in that order.

What is speculative decoding and how much faster is it?

Speculative decoding uses a small draft model to propose tokens while the big model verifies several at once, turning sequential generation into parallel verification. It typically gives 1.5–3x speedup on GPU without changing the output distribution.

Is it worth using two GPUs or CPU+GPU together?

For models that barely fit one GPU, offloading part to CPU or splitting across two GPUs lets you run them at all — often 5–15 t/s. For models that already fit, adding hardware rarely helps; a smaller or better-quantized model is the smarter move.