Why Text-Only RAG Misses So Much
A contract with a signed table of values, a slide deck, a flow diagram, a screenshot of a dashboard — none of these exist as text in your corpus. Classic RAG either skips them or indexes only filenames, which means queries about what's inside the image return nothing. For document-heavy domains (legal, medical, enterprise), that's not a corner case — it's most of the corpus.
The Multimodal Pipeline, Layer by Layer
- Parse. Layout-aware parsing splits pages into text blocks, tables, and figures — PyMuPDF or unstructured keep the structure (this is document-aware chunking applied to PDFs).
- OCR. Scanned pages go through OCR (Tesseract or a local OCR model) to produce the text layer. Tables may need specialized table-extraction.
- Describe. Each figure/screenshot gets a VLM-generated caption — "Line chart showing revenue rising 12% from Q1 to Q2 2026, with a note..." — stored as text alongside the image reference.
- Embed + index. Text chunks and image descriptions both go through your embedding model into ChromaDB/FAISS, each tagged with source + page metadata.
- Retrieve + answer. Queries search the combined index; when a figure is retrieved, the VLM can answer directly about the image at generation time.
# The describe step, locally
from vllm import LLM # or Ollama with a VLM
vlm = LLM(model="Qwen/Qwen-VL") # caption images at ingest
def caption(image_path):
prompt = ("Describe this image factually: what it shows, "
"its data, labels, and conclusions. For search indexing.")
return vlm.generate([...], prompt)Embed Images vs Describe Them: The Decision
| Approach | How it works | Best when |
|---|---|---|
| Describe-then-embed ⭐ | VLM caption → embed the text | Default — works with any vector store, cheap queries, human-readable index |
| Multimodal embeddings | Images + text projected into one space | Image-to-image search, visual similarity, "find similar charts" |
| VLM at query time | Retrieve a reference, let the VLM read the actual image | When the answer requires reading the image itself (a number, a label) |
The honest guidance: start with describe-then-embed. It's simple, works with your existing RAG pipeline, and covers most queries. Add query-time VLM reading when users ask about specific visual details. Multimodal embeddings are the advanced option for genuine visual search.
Local Models That Make It Work (2026)
| Job | Local pick |
|---|---|
| Unified multimodal LLM | Gemma 4 12B (June 2026 — screenshots, documents, UI); Qwen3-VL family |
| Document/screenshot extraction | LFM2.5-VL extract models (450M / 1.6B — tiny, purpose-built) — see the LFM guide |
| OCR | Tesseract (free), or local OCR models for lower quality scans |
| Layout parsing | PyMuPDF, unstructured (open source) |
| Embeddings | BGE-M3 or any model from the ranking |
The Cost Reality
- Index time is the bill. OCR + one VLM caption per image. A few thousand images caption overnight on a laptop; on a 12GB GPU it's hours.
- Query time stays cheap — search runs on text embeddings, same as classic RAG.
- Cache parsed text so re-chunking doesn't re-OCR — the parsing layer is the expensive, stable one.
- For regulated domains (legal/medical), every caption is derived data — keep the source image reference so answers can cite the original page. GDPR-style documentation applies to the derived index too.
Frequently Asked Questions (FAQ)
What is multimodal RAG?
Multimodal RAG extends classic RAG to non-text content: scanned PDFs, screenshots, diagrams, charts, and tables. It typically works by extracting text (OCR), describing or captioning visual content with a vision model, and indexing both — so retrieval can find the information inside images, not just around them.
Do I need a vision-language model for multimodal RAG?
For most pipelines, yes — a VLM generates descriptions/captions of images and answers visual questions, which get indexed as text. Pure image-embedding models exist but are less mature for retrieval. In 2026, Gemma 4 (12B) and Qwen-VL-class models run this locally.
What is the best way to handle scanned PDFs?
The reliable stack: layout-aware PDF parsing (PyMuPDF or unstructured) → OCR for scanned pages (Tesseract or a local OCR model) → keep the text with page/section metadata → chunk and embed. The text layer is what retrieval searches; images inside PDFs get described separately.
How do I index images for retrieval?
Two patterns: (1) generate a text caption/description with a VLM and embed that text — simple and works with any vector store; (2) use a multimodal embedding model that projects images and text into the same space, enabling image-to-image and text-to-image search. Start with (1).
Is multimodal RAG expensive?
Indexing is the cost — OCR and VLM captioning are compute-heavy at ingest time (one VLM call per image). Query time stays cheap since search runs on text embeddings. On local hardware, caption a few thousand images overnight; the pipeline itself runs on a laptop.
What models support multimodal RAG locally in 2026?
Gemma 4 12B (unified multimodal, June 2026), Qwen-VL / Qwen3-VL family, and LFM2.5-VL extract models (450M and 1.6B, purpose-built for document/screenshot extraction). For OCR specifically, Tesseract and local OCR models remain the workhorses. All run on consumer hardware.