How to Build a Document Q&A System That Cites Its Sources (for Law Firms)

Why Citations, and Why Local

An uncited AI answer is an assertion. A cited answer is a finding. That distinction is the whole game for legal work, and it maps onto two obligations lawyers already have.

Confidentiality. The moment a client document touches a cloud chatbot, a third party holds the very material that may be protected by attorney–client privilege — the pattern a federal court found defeats privilege in United States v. Heppner (S.D.N.Y. 2026). ABA Formal Opinion 512 is explicit that lawyers must protect client confidences when using generative AI under Model Rule 1.6. Running the pipeline on your own hardware is the strongest architectural answer: there is no moment where the document exists anywhere but your disk.

Competence. Opinion 512 also requires competence: understand the tool's limits and verify its output. A system that answers from memory makes verification impossible — you can't check a citation that doesn't exist. A system that answers from retrieved passages and shows the source makes verification a two-second operation. For the full ethics picture, see Is It Ethical to Use ChatGPT for Contract Review? and the plain-English privilege guide.

The Stack

This is the same architecture the open-source Lawyer Assistant uses — embeddings, vector store, keyword index, and LLM running as separate local services so each piece can be swapped without touching the rest.

Setup

# Pull the models once (Ollama)
ollama pull bge-m3          # embeddings
ollama pull qwen3:8b        # generation

# Python
pip install langchain langchain-community \
            langchain-chroma langchain-ollama \
            chromadb pypdf rank-bm25

Step 1: Load and Chunk Legally

Contracts and briefs are not Wikipedia articles. A definition in Section 1 governs a term in Section 12, and a clause spans paragraphs. So chunking is a first-class decision: split on clause boundaries first, and carry source and page metadata through every chunk — that metadata is what becomes the citation later.

from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter

loader = PyPDFLoader("merger-agreement.pdf")
pages = loader.load()  # one Document per page, page number in metadata

splitter = RecursiveCharacterTextSplitter(
    chunk_size=600,
    chunk_overlap=80,
    # legal text breaks on clause boundaries before sentences
    separators=["\n\n", "\n", ". ", "; ", " ", ""],
)
chunks = splitter.split_documents(pages)

for i, chunk in enumerate(chunks):
    chunk.metadata["doc_id"] = f"c{i}"  # stable id for fusion

print(f"{len(chunks)} chunks")
print(chunks[0].metadata)  # {'source': 'merger-agreement.pdf', 'page': 1, 'doc_id': 'c0', ...}

If your PDFs are scans, run OCR before this step — Lawyer Assistant does this automatically so scanned exhibits are searchable too. For the full trade-off between fixed-size, document-aware, and contextual chunking, see Advanced RAG: Chunking Strategies Compared.

Step 2: Embed and Store

Each chunk gets embedded with BGE-M3 and stored in ChromaDB, persisted to disk. Re-run and you load instead of re-embedding.

from langchain_ollama import OllamaEmbeddings
from langchain_chroma import Chroma

embeddings = OllamaEmbeddings(model="bge-m3")

vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    ids=[c.metadata["doc_id"] for c in chunks],
    persist_directory="./legal_index",
)
# prove it works: the store answers a similarity query immediately
probe = vectorstore.similarity_search("indemnification cap", k=1)
print("indexed", len(chunks), "chunks; probe returned",
      probe[0].metadata["doc_id"])

Step 3: Hybrid Retrieval — BM25 + Vectors, Fused

Legal language is unforgiving in both directions. A contract clause may use exact boilerplate phrasing that pure semantic search would paraphrase away; a casual question may use words that never appear in the document. So run two retrievers in parallel and merge the rankings with reciprocal rank fusion (RRF):

• BM25 catches exact terms, clause citations, and legal shorthand
• Vector search catches meaning, so "what happens if the other party breaches" finds a clause about "default by counterparty"

from rank_bm25 import BM25Okapi

tokenized = [c.page_content.lower().split() for c in chunks]
bm25 = BM25Okapi(tokenized)


def hybrid_search(query: str, k: int = 6, rrf_k: int = 60):
    # 1) semantic ranking — nearest chunks first
    vec_hits = vectorstore.similarity_search_with_score(query, k=k)
    vec_ids = [d.metadata["doc_id"] for d, _ in vec_hits]

    # 2) keyword ranking — BM25 over the same chunks
    scores = bm25.get_scores(query.lower().split())
    bm25_ids = [chunks[i].metadata["doc_id"]
                for i in scores.argsort()[::-1][:k]]

    # 3) reciprocal rank fusion — merge both orders into one
    fused = {}
    for rank, doc_id in enumerate(vec_ids + bm25_ids):
        fused[doc_id] = fused.get(doc_id, 0.0) + 1.0 / (rrf_k + rank + 1)
    top = sorted(fused, key=fused.get, reverse=True)[:k]

    by_id = {c.metadata["doc_id"]: c for c in chunks}
    return [by_id[d] for d in top]


for d in hybrid_search("What happens if the buyer breaches closing?"):
    print(d.metadata["doc_id"], d.metadata["page"], d.page_content[:60])

RRF needs no score normalization — it just fuses ranks, which is why it works when BM25 scores and embedding distances are on totally different scales. For the deeper reasoning, see What Is Hybrid Search? BM25 + Semantic Retrieval. If precision matters more, add a cross-encoder reranker over the top-k — the pattern is in Reranking in RAG.

Step 4: Cited Generation

Now the part that makes answers verifiable. The retrieved passages are numbered and passed as grounding context; the prompt instructs the model to answer only from them and to end every factual claim with a citation carrying the passage's real file name and page. Generation runs at temperature 0.

from langchain_ollama import ChatOllama
from langchain_core.prompts import ChatPromptTemplate

llm = ChatOllama(model="qwen3:8b", temperature=0)

PROMPT = """You are a legal research assistant working from a document set.
Answer using ONLY the numbered passages below. For every factual claim,
end the sentence with a citation in this exact form:
[Source: {{source}}, p.{{page}}]
Use the passage's real file name and page number. If the passages do not
contain the answer, say "The documents do not address this." Never invent
a citation.

Passages:
{context}

Question: {question}"""


def format_context(docs):
    blocks = []
    for i, d in enumerate(docs, 1):
        src = d.metadata.get("source", "unknown")
        page = d.metadata.get("page", "?")
        blocks.append(f"[{i}] ({src}, p.{page})\n{d.page_content}")
    return "\n\n".join(blocks)


prompt = ChatPromptTemplate.from_template(PROMPT)
chain = prompt | llm

question = "What happens if the buyer breaches the closing obligations?"
answer = chain.invoke(
    {"context": format_context(hybrid_search(question)), "question": question}
)
print(answer.content)

Note the doubled braces around {{source}} and {{page}} — that's how LangChain templates emit a literal citation format to the model while keeping {context} and {question} as real variables.

Step 5: Verify the Citations

Prompting is not enough — citation-shaped hallucinations are a known failure mode. So verify mechanically: parse every [Source: X, p.N] marker and check it against the passages that were actually retrieved. Anything that doesn't match is flagged. This is a contract, not a suggestion: if the passage doesn't exist, the answer doesn't ship.

import re


def verify_citations(answer, docs):
    markers = re.findall(r"\[Source:\s*(.+?),\s*p\.(\d+)\]", answer)
    real = {(d.metadata.get("source"), str(d.metadata.get("page"))) for d in docs}
    invented = [(s, p) for s, p in markers if (s, p) not in real]
    print(f"{len(markers) - len(invented)}/{len(markers)} citations "
          "trace to retrieved passages")
    if invented:
        print("INVENTED:", invented)
    return invented


verify_citations(answer.content, hybrid_search(question))

Step 6: Measure It, Don't Trust It

Before this system earns a place in a real workflow, measure it the way you'd measure any legal tool: on a fixed test set. Build ~50 questions with known answers from your own documents, then score the pipeline with RAGAS on the two metrics that matter for legal work:

• Faithfulness — does the answer stay within the retrieved passages, or does the model drift into memory?
• Answer relevancy — does it actually address the question asked?

Pair that with the manual spot-check: take ten answers and open every citation. If a page doesn't match, the system isn't grounded, no matter how confident it sounds. The full methodology — building the eval set, setting thresholds, and deciding when a system is trustworthy — is in How to Evaluate a RAG System, with the metric definitions in RAG Evaluation.

What a Firm Adds Before Production

1. Access control — who may query which matters; privilege and confidentiality flow from who can reach the data
2. Audit logging — every question, answer, and opened citation logged, so usage is reviewable
3. A written AI policy — approved tools, prohibited tools, and review procedures; under Model Rule 5.1, firm leadership is responsible for supervision
4. Human review — AI triages and cites; a lawyer decides. Every answer crosses a desk before it shapes advice
5. Client communication — where required, informed consent about how AI is used on the matter

Frequently Asked Questions (FAQ)

What does it mean for a document Q&A system to cite its sources?

The system retrieves the relevant passages from your own documents first, then generates the answer using only those passages — and appends a citation in the form [Source: file, p.N] to every claim. Each citation points at the exact passage the model relied on, so a lawyer can open the source and verify before relying on it.

Why hybrid retrieval (BM25 + vectors) for legal documents?

Legal language is both exact and paraphrased. A clause may use boilerplate phrasing that a pure semantic search would paraphrase away, while a casual question may use words that never appear in the document. BM25 catches exact terms, clause citations, and legal shorthand; vector search catches meaning and paraphrase. Fusing both with reciprocal rank fusion beats either alone.

Does this run on a normal laptop, and is it private?

Yes. The whole pipeline — BGE-M3 embeddings, ChromaDB, BM25, and a quantized 7–14B model via Ollama — runs on a modern laptop with 16GB of RAM. Documents, embeddings, and answers never leave the machine, which is the strongest position you can take on attorney–client confidentiality (Model Rule 1.6).

How do you stop the model from inventing citations?

Three layers: the prompt is constrained to answer only from numbered retrieved passages and to cite the passage's real file name and page; generation runs at temperature 0; and a verification step parses every [Source: X, p.N] marker and checks it against the passages that were actually retrieved, flagging any that don't exist.

How do you measure whether the system is trustworthy?

Build a small evaluation set of questions with known answers, then measure faithfulness (does the answer stay within the retrieved passages?) and answer relevancy (does it address the question?) using a framework like RAGAS. Pair that with a manual spot-check: take ten answers and open every citation.

Can I just use ChatGPT for this?

For drafting, yes. For answering from client documents, no: pasting privileged material into a public chatbot exposes it to third parties — the pattern courts found defeats privilege in United States v. Heppner (S.D.N.Y. 2026) — and the model answers from memory, not from your documents. A local, cited RAG system solves both problems.

Sources & Further Reading

• ABA Formal Opinion 512: Generative Artificial Intelligence Tools (PDF) — the competence, confidentiality, and supervision duties for lawyers using AI
• LangChain RAG tutorial (official docs)
• RecursiveCharacterTextSplitter documentation
• BGE-M3 — multilingual embedding model (Hugging Face)
• ChromaDB documentation
• Ollama — run LLMs locally
• rank-bm25 (PyPI)
• RAGAS — evaluation framework for RAG pipelines