Phase 1: Decide and Size
- Confirm the case: data-residency requirement, confidentiality, cost at your volume, or offline reliability — from the risk analysis. If none apply, cloud may honestly be the better buy.
- Pick model + quant: 14B Q4_K_M for a team workload, 30B+ for quality-critical. Read sizes in the RAM guide; check licenses in the open-model ranking.
- Budget memory: VRAM ≥ file size + KV cache + overhead (rule of thumb: 1.5–2x the GGUF size at 8–16K context); system RAM ≥ file size for offload headroom. Add ~30% for growth.
- Choose the machine: one 24GB GPU → 14B; two 24GB or one 48GB+ → 30B; four 24GB → 70B-class. Budget paths exist.
Phase 2: Serve
# The production path: vLLM
vllm serve Qwen/Qwen3-14B-GGUF \
--quantization gguf \
--max-model-len 32768 \
--gpu-memory-utilization 0.9 \
--port 8000
# Team/dev path: Ollama (see the Ollama vs vLLM guide
# for when to graduate)- Pin everything: vLLM version, model revision (SHA), quant. Reproducibility is a compliance feature.
- Load-test before go-live: concurrent users × tokens/sec vs your target. The Ollama vs vLLM guide explains why throughput matters under concurrency.
- Set a context policy: max-model-len is a hard budget — enforce it at the API so long prompts fail cleanly instead of OOMing.
Phase 3: Secure
| Control | Implementation |
|---|---|
| Network | Internal VLAN only; no public exposure; egress restricted (models are local — nothing needs to phone home) |
| Auth | API keys or SSO in front of the serving endpoint; role-based access |
| Transport | TLS internally; mTLS if the network segment warrants it |
| Logging | Request logs with a PII scrubber; retention per your data policy |
| Integrity | Hash-verify model files at load; pin revisions |
| Patches | OS, runtime, and driver updates on a cadence — the model isn't the attack surface, the server is |
Phase 4: Monitor and Operate
- Metrics that matter: GPU utilization, VRAM usage, tokens/sec, queue depth, TTFT (time to first token), inter-token latency, error rate.
- Alert on: VRAM near-exhaustion (the #1 prod failure — causes OOM reloads that take the service down), latency regressions, model crash loops.
- Backups: model files are static (easy), but configuration, logs, and any vector stores need the same discipline as databases.
- Capacity review quarterly: usage grows; plan the next GPU before the queue becomes the bottleneck.
🚀 The pattern in practice
This is exactly how Lawyer Assistant-style deployments run: local model + vLLM + ChromaDB behind the firm's own perimeter, with everything in this checklist — because when the deployment is inside a regulated environment, ops discipline is the compliance story. See regulated industries and on-premise serving for the wider context.
Frequently Asked Questions (FAQ)
When does on-premise LLM deployment make sense?
When data cannot leave your infrastructure (regulated industries, client confidentiality), when per-token cloud costs at your volume exceed hardware, when you need offline reliability, or when auditability requires full control of the stack. The honest counter-case: small teams with modest data sensitivity often don't need it.
How do I size hardware for an on-premise LLM?
Pick the model + quant, read the size from the RAM-ranked guide, then budget: VRAM ≥ weights + KV cache + overhead (typically 1.5–2x the GGUF file size for reasonable context); system RAM ≥ weights for offload headroom; add 30% for growth. One 24GB GPU serves a 14B model to a team; 30B+ needs 2x 24GB or a 48GB+ card.
What should I use to serve the model in production?
vLLM is the default production server in 2026 — continuous batching for throughput, PagedAttention for memory, OpenAI-compatible API, and mature observability. Ollama is fine for small teams and dev. The Ollama vs vLLM guide in this blog has the full comparison.
How do I secure an on-premise LLM server?
Same discipline as any internal service: network segmentation (no public exposure), authentication on the API, TLS internally, role-based access, prompt/request logging, model file integrity checks, and a patch cadence. The model is not the attack surface — the server and its data are.
What should I monitor?
GPU utilization, VRAM usage, tokens/sec, queue depth, request latency (TTFT + inter-token), error rate, and model health. Log prompts with a privacy scrubber. Alert on VRAM exhaustion (it causes OOM reloads) and latency regressions. Dashboards from Prometheus/Grafana or vLLM's built-in metrics cover it.
What are the common failure modes?
VRAM exhaustion under concurrent load, context overflow on long prompts, model reload storms after crashes, quant quality surprises in production, and silent version drift of the serving stack. Each has a mitigation in this checklist — test, pin, and monitor.