The Difference That Changes Everything
An API-based AI developer optimizes for tokens and service costs — their model runs in someone else's data center, and your confidential documents travel to reach it. An offline AI engineer optimizes for hardware and boundaries — the model runs on machines you own, inside your network, and your data never leaves.
That inversion changes the entire skill set. If your use case is confidential — legal work, healthcare, finance, or anything under GDPR or corporate policy — the second profile isn't a luxury. It's the only one that can legally build it. (For the legal stakes, see what happened when a court found AI-generated legal documents unprotected.)
The Five Skills That Matter
🧮 Hardware sizing math
Can they size VRAM and RAM from a model's parameter count and quantization level? A 7B model fits one 24 GB GPU; a 70B model needs multiple 80 GB GPUs. If they can't do this math from memory, they've never deployed.
🗜️ Quantization fluency
GGUF, GPTQ, AWQ — which to use when, and the quality/size trade-off of each. This is the difference between "the model won't fit" and "here's the 8-bit version that does." See GGUF vs AWQ vs GPTQ.
🔌 Local serving and operations
Experience running a serving engine (vLLM, llama.cpp, NVIDIA NIM) on your own infrastructure — plus updates and model swaps without internet access. Air-gapped operation is a whole skill of its own; see Air-Gapped AI.
📚 On-premise RAG
Retrieval pipelines and vector databases deployed inside your boundary, with answers that cite their sources — not a cloud vector store that sends your documents somewhere else.
🧪 Evaluation on your data
They test on your documents, score faithfulness and relevance separately, and set thresholds before calling it done. If they only show demos, they haven't evaluated anything.
How to Test for It: The Interview Questions
| Ask | What a strong answer sounds like | Red flag |
|---|---|---|
| "Tell me about a model you actually deployed on your own hardware." | A specific model, specific hardware, quantization choice, and what broke | Only mentions API calls or tutorials |
| "Which model fits a 16 GB MacBook for document Q&A, and why?" | A concrete model + quantization + context-length trade-off | "You should use the cloud for that" |
| "How do you update a model in an air-gapped environment?" | Offline artifact transfer, checksums, staged rollout | No answer — updates never crossed their mind |
| "How do you prove the answers are correct?" | Test set, faithfulness/relevance scores, thresholds, citations | "The model is very accurate" |
| "What happens when your system hallucinates a legal or medical answer?" | Grounding, refusal strategy, human-in-the-loop, monitoring | "That won't happen" |
💡 The one-question shortcut: ask them to walk through a project where data could not leave the building. The candidate who has done this will light up; the API-wrapper developer will go quiet. Everything else is detail.
The Mindsets That Predict Success
- Security by default. They mention access control, audit logs, and encryption before you ask — not as an add-on.
- Honesty about limits. They tell you what a local model can't do on your hardware before selling you what it can. (See How Much RAM Do You Need to Run Local AI? for the honest baseline.)
- Ownership of the last 20%. Production hardening, monitoring, and maintenance are part of their pitch, not a change order.
- Plain-language communication. They can explain quantization or retrieval to a non-technical decision-maker without jargon.
The broader hiring context — pricing, scope, red flags, pilots — is in Hiring an AI Developer? 10 Questions to Ask Before You Start.
Frequently Asked Questions (FAQ)
What does an offline AI engineer do?
An offline (or on-premise) AI engineer builds AI systems that run entirely inside your network boundary: model selection, hardware sizing, quantization, serving, retrieval, and evaluation — with no data ever sent to a third-party API. It is a systems-engineering role first and an AI role second.
What is the difference between an AI developer and an offline AI engineer?
Most AI developers build on third-party APIs — the model lives in someone else's data center and your data crosses their network. An offline AI engineer runs models on hardware you own: they must size GPUs and RAM, quantize models, operate serving engines, handle updates without internet, and keep everything inside your security boundary.
What skills should I look for when hiring for private AI?
Look for: model selection and quantization (GGUF/GPTQ/AWQ), GPU and RAM sizing math, local serving engines, RAG and vector databases deployed on-premise, evaluation on your own documents, security and access control, and experience running systems without internet access. A candidate should be able to talk about a model they actually deployed, not just a prompt they wrote.
Why can't I just use an API-based AI developer for private AI?
Because the core constraint is inverted. API developers optimize for tokens and costs of a service; private AI optimizes for hardware, quantization, and staying inside your network. Sending confidential data to a third-party API can violate legal privilege, HIPAA, GDPR, or corporate policy — the whole point is that the data never leaves.
Do I need an offline AI engineer if I already have an IT team?
An IT team can host servers; private AI additionally needs LLM-specific expertise: which model fits your hardware, how to quantize it, how to serve it reliably, how to build retrieval that cites sources, and how to evaluate accuracy. If your IT team has no LLM deployment experience, the engineer fills that gap.
🔍 Building private AI and need the right engineer?
I design and deploy privacy-first local AI systems — on-premise LLMs, private RAG with cited answers, and multilingual assistants — through Haal Lab. Contact me for a scoping conversation, no obligation.