Contextual Retrieval: Chunking That Carries Context (2026)

The Problem: Ambiguous Chunks

Chunking (covered in the strategies guide) cuts documents into pieces — and pieces lose context. A chunk reading "the parties agree to extend the term by 12 months" is unusable to retrieval unless the model knows it's about the supply agreement's renewal clause. Embeddings capture similarity, not provenance. When many chunks look alike, the query matches all of them equally — and the right one gets lost.

How Contextual Retrieval Works

Chunk as usual. Any strategy from the chunking guide.
Generate context. For each chunk, ask an LLM: "Where does this chunk sit in the document, what precedes and follows it, and what is it about? Write it so the chunk can be understood alone."
Prepend context. Store context + chunk text as the unit.
Embed the enriched unit — and index the enriched text for BM25 too, if you use hybrid search.

CONTEXT_PROMPT = """<document>
{document}
</document>
The chunk below is from the document above. Write 2-3
sentences of context: which section it belongs to, what
surrounds it, and what it's about — so the chunk can be
understood completely on its own."""

context = llm.generate(CONTEXT_PROMPT.format(chunk=chunk_text))
enriched = f"{context}\n\n{chunk_text}"
embedding = embed_model.encode(enriched)  # ← the fix

The Evidence: 49% Fewer Failures

Anthropic's September 2024 evaluation on a codebase + business documents corpus measured top-20 retrieval failures:

Setup	Failure reduction vs baseline
Plain embeddings (baseline)	—
Contextual embeddings	−49%
Contextual embeddings + hybrid search	−67%
+ Reranking on top	Further gains (see reranking guide)

The mechanism is simple: the context block turns "extend the term" into "[Supply Agreement, Renewal Clause §5.2: the parties agree to extend the term by 12 months]" — an unambiguous, query-matchable unit. This compounds with every other retrieval lever; it doesn't replace them.

Running It Locally (Free)

Context generation: any sub-12B model via Ollama writes good context blocks. Qwen3-8B or Qwen3.5-9B are solid; even a 4B handles the prompt.
Cost shape: one call per chunk, one-time. 10,000 chunks ≈ an evening on a laptop, minutes on a GPU. Use speculative decoding to speed the batch.
Storage: the enriched text is a bit bigger — trivial for ChromaDB/FAISS.
Privacy: everything stays on your machine — a natural fit for offline workspaces and document assistants.

Prove It on Your Corpus

Build a 30–50 question eval set (the evaluation guide shows how).
Measure baseline recall@k on plain embeddings.
Re-index with contextual blocks; re-measure.
If recall moved meaningfully, keep it. If not, your chunks were already self-contained — spend the effort elsewhere.

💡 Where it shines most: long, repetitive, section-heavy documents — contracts, manuals, filings — where identical phrasing repeats across contexts. If your corpus is short standalone notes, contextual retrieval buys less. Measure first.

Frequently Asked Questions (FAQ)

What is contextual retrieval?

Contextual retrieval is a technique where each chunk gets a short, LLM-generated context block — where it comes from, what surrounds it, what it's about — before being embedded. The embedded chunk becomes self-contained, so a query can match it even when the chunk text alone is ambiguous. Anthropic's 2024 research measured a ~49% cut in retrieval failures.

Does contextual retrieval really cut retrieval failures by half?

Anthropic's controlled evaluation found contextual embeddings reduced top-20 retrieval failures by 49%, and by 67% when combined with hybrid search (BM25 + embeddings). Exact numbers vary by corpus, but the effect is consistently large — the biggest single chunking-level win in recent years.

How much does contextual retrieval cost?

One LLM call per chunk at index time (plus the embedded context tokens). For a few thousand chunks, that's trivial; for millions it's a real indexing budget. Query-time cost barely changes. Locally, a sub-12B model generates the context blocks for free.

What context should the LLM write for each chunk?

The standard prompt asks for: which document and section the chunk comes from, what the chunk is about, and what precedes/follows it — written so the chunk could be understood completely on its own. Keep it short (2–3 sentences); the point is disambiguation, not summarization.

Does contextual retrieval help with hybrid search too?

Yes — the technique applies to both sides. Contextual BM25 indexes the chunk text + context for keyword matching, and contextual embeddings do the same for semantic matching. The two compound: Anthropic measured 49% (embeddings) and 67% (combined) failure reductions.

When should I NOT use contextual retrieval?

When chunks are already self-contained (short standalone documents, single-purpose notes), when indexing budget is extremely tight, or when your eval shows it doesn't move your metrics. Always measure with a small eval set first — see the RAG evaluation guide.