Advanced RAG: Chunking Strategies Compared (2026 Guide)

The Five Strategies Compared

Strategy	How it splits	Cost	Best for
Fixed-size	N tokens per chunk, with overlap	Free	Baseline, homogeneous text
Recursive	Hierarchical separators (headings → paragraphs → sentences)	Free	Default starting point
Semantic	Embedding-similarity drops between sentences	Embedding compute	Prose with shifting topics
Document-aware	Structure: headings, code blocks, tables, sections	Parser work	Markdown, code, PDFs, HTML
Contextual	Any base split + LLM-generated context per chunk	LLM calls per chunk	When retrieval quality matters most

1. Fixed-Size Chunking (The Baseline)

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,      # tokens
    chunk_overlap=50,    # ~10% overlap
    separators=["\n\n", "\n", ".", " ", ""],
)
chunks = splitter.split_text(document)

Simple, predictable, and a fine baseline — but it cuts wherever the size limit lands, splitting sentences and even thoughts. Use it to establish a baseline score, then move up. Always include overlap (10–20%): it's the cheapest recall improvement you'll ever make.

2. Semantic Chunking (Split on Meaning)

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-m3")

def semantic_chunks(sentences, threshold=0.35):
    emb = model.encode(sentences, normalize_embeddings=True)
    sims = np.array([np.dot(emb[i], emb[i+1])
                     for i in range(len(emb)-1)])
    breaks = np.where(sims < threshold)[0] + 1
    return np.split(sentences, breaks)

Embed each sentence, and start a new chunk where similarity between neighbors drops sharply — that's where the topic changes. Produces far more coherent chunks, at the cost of one embedding pass during indexing and a tunable threshold. Works beautifully with good embedding models like BGE-M3.

3. Document-Aware Chunking (Use the Structure)

# Markdown: split on headings, keep hierarchy as metadata
from langchain_text_splitters import MarkdownHeaderTextSplitter

splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=[("#", "H1"), ("##", "H2"), ("###", "H3")]
)
chunks = splitter.split_text(markdown_doc)

# Code: split on function/class boundaries
from langchain_text_splitters import RecursiveCharacterTextSplitter
python_splitter = RecursiveCharacterTextSplitter.from_language(
    language="python", chunk_size=500, chunk_overlap=50
)

Documents arrive with structure — use it. Markdown chunks by heading, code by function/class, PDFs by layout-parsed sections (with a tool like unstructured or PyMuPDF). Each chunk can carry metadata ({"section": "Chapter 3", "page": 41}) that filters and re-ranks retrieval. Most "my RAG is bad" problems dissolve at this step.

4. Contextual Retrieval (The Big Recent Win)

prompt = """<document>
{chunk_text}
</document>
Write a short, self-contained context block explaining
where this chunk appears in the document and what it is
about, so a search query can find it on its own."""

context = llm.generate(prompt)
final_chunk = f"{context}\n\n{chunk_text}"
embedding = embed_model.encode(final_chunk)

Anthropic's 2024 contextual retrieval research showed the pattern: before embedding, have an LLM write a short context block for each chunk — what document it's from, what surrounds it, what it's about. Embedded with that context, chunks become self-describing, and retrieval failures dropped ~49% (and up to 67% combined with hybrid search).

The cost is real — one LLM call per chunk at index time — which is why it's the advanced option. But for privacy-first legal documents where a missed clause is a failure, the quality jump is worth every call. Our contextual retrieval guide goes deeper.

Decision Guide: Which Strategy When

Your corpus	Start with	Upgrade to
Clean markdown / docs	Recursive (paragraph-aware)	Document-aware (headings)
Code repositories	Language-aware splitter	Function/class boundaries + symbol metadata
Scanned PDFs / reports	Layout-aware parser first	Section chunks + page metadata
Mixed prose, shifting topics	Recursive	Semantic chunking
Anything where accuracy is critical	Document-aware	Contextual retrieval + hybrid search

💡 The workflow: build a 20–50 question eval set from real user queries, chunk your corpus two ways, and compare recall@k. The strategy that wins your eval is your strategy — this is the only honest way to pick chunk size and method.

Practical Rules That Survive Any Method

Overlap 10–20% — always, regardless of strategy.
Keep metadata — source, section, page, date. Retrieval filters and answer citations need it.
Match chunk size to your embedding model's context — embedding models truncate; see the ranking for context lengths.
Chunk per document type — one strategy for the whole corpus is a compromise, not a design.
Re-chunk is cheap, re-embed is not — cache parsed text so you can re-split without re-parsing PDFs.
Reranking fixes boundary misses — pair chunking with cross-encoder reranking before blaming the splitter.

Frequently Asked Questions (FAQ)

What is the best chunk size for RAG?

There is no universal best size — it depends on your content and embedding model. 200–500 tokens with 10–20% overlap is a strong starting point. Short chunks retrieve more precisely; long chunks give the model more context. Benchmark your own corpus with a small eval set rather than copying someone else's number.

What is semantic chunking?

Semantic chunking splits text at points where meaning changes — measured by embedding similarity between consecutive sentences. When similarity drops sharply, a new chunk starts. It produces more coherent chunks than fixed-size splitting but costs more compute and adds latency during indexing.

Should I use overlap between chunks?

Yes — 10–20% overlap prevents sentences from being split mid-thought and helps recall around boundaries. The cost is a small amount of duplicated storage and slightly more retrieved noise. Disable overlap only when storage is genuinely tight.

What is contextual chunking / contextual retrieval?

Contextual retrieval adds a short context block to each chunk — where it came from, what surrounds it — often generated by an LLM, before embedding. Anthropic's 2024 research showed this can cut retrieval failures roughly in half, making it the biggest single chunking win in recent years.

Does chunking strategy actually change retrieval quality?

Dramatically. The same corpus can go from unusable to reliable by switching from naive fixed-size chunks to document-aware or contextual chunking. Chunking is routinely the first thing to fix when a RAG system retrieves irrelevant context.

What is the best chunking for code, PDFs, and markdown?

Use document-aware chunking: split on structure. Code should chunk by function/class boundaries, markdown by headings, PDFs by sections with layout-aware parsing. Structure is free metadata — most RAG quality problems come from ignoring it.