Production RAG for Legal Research: The JurIA Playbook

A chatbot that answers legal questions is a party trick. A retrieval system that answers with citations you can verify is a product. JurIA — the legal-research assistant I'm building for the Moroccan market — is deliberately the second kind. This is the playbook so far, including the mistakes.

The problem worth solving

Legal research in Morocco is painful for structural reasons. The source texts live in the Bulletin Officiel, scattered codes, and OHADA conventions. They are multilingual (French and Arabic, with legal terms that don't translate cleanly), fragmented across portals, and rarely annotated or normalized. Lawyers, researchers, and compliance teams spend hours cross-referencing documents; search portals return documents, but nobody returns answers.

That gap — between "here are ten documents" and "here is the answer and the article that supports it" — is where grounded retrieval earns its keep.

Corpus engineering comes first

Before any model touches the text, the corpus has to be made retrievable. Legal text does not chunk like prose. A statutory article is a self-contained unit of meaning; a judgment is a narrative; a regulation references other regulations. Naive fixed-size chunking (say, 512 tokens) cuts articles in half and produces neighbors that answer nothing.

The rule that emerged:

• Chunk at semantic boundaries. For codes, chunk per article. For judgments, chunk per section. For regulations, keep the hierarchy — part, chapter, article — in the chunk metadata.
• Carry context with the chunk. Each chunk stores its document, jurisdiction, language, and section path. That metadata is not decoration; it is the retrieval filter.

def chunk_legal_document(doc):
    chunks = []
    for section in split_by_structure(doc):
        chunks.append({
            "text": section.text,
            "meta": {
                "jurisdiction": section.jurisdiction,  # MA | FR | OHADA
                "doc_type": section.doc_type,          # code | judgment | regulation
                "path": section.path,                  # Book > Title > Article
            },
        })
    return chunks

Hybrid retrieval with a reranker

Dense embeddings alone fail on legal text. Legal language is formulaic — "nonobstant", "au cas où", "sous réserve de" — and synonyms are lethal: the correct article uses terms the question never does. Keyword search catches exact statutory language; dense search catches paraphrase; neither alone is enough.

JurIA uses hybrid retrieval over Milvus:

1. Dense pass with embeddings trained (or fine-tuned) for French and Arabic legal text.
2. Sparse/keyword pass with the statutory terms weighted by document frequency.
3. Score fusion, then a cross-encoder reranker over the top-40 candidates to produce the final top-5.

The reranker is the most underrated component in the stack. Retrieval gets you 40 plausible candidates; the reranker decides which three actually answer the question. In legal RAG, precision at the top is non-negotiable — a wrong statute quoted confidently is worse than no answer at all.

Citations are the product

The generation step has one hard rule: the model can only answer from retrieved passages, and every claim carries its source. Anything ungrounded is refused, not guessed. This is enforced structurally, not by prompt. The prompt includes only the retrieved passages; the citation objects are attached to the answer from the retrieval metadata, not generated by the model.

// every answer carries machine-readable citations
{
  answer: "Under Article X of the Moroccan Civil Code…",
  citations: [
    { article: "X", source: "C.civ., art. X", jurisdiction: "MA" },
  ],
}

What this buys in practice: a lawyer can verify an answer in under a minute, and a hallucinated statute cannot be cited because the citation comes from the chunk metadata, not from the token stream.

Jurisdiction as a first-class filter

Morocco, France, and OHADA are different legal systems with different sources. Conflating them is worse than returning nothing — a French code article looks plausible and is simply wrong law in Morocco. So jurisdiction is not a search refinement; it is a hard filter applied before retrieval, and it is surfaced in the UI as a selector the user controls explicitly.

Evaluation: the boring engine of trust

The accuracy bar for legal work is unforgiving. We evaluate on three axes:

• Retrieval quality: recall@k against a gold set of question→article pairs.
• Groundedness: the fraction of answer claims supported by the cited passage (checked with an LLM-judge plus human review on a sample).
• Legal correctness: reviewed by people who practice law, not by people who write RAG blog posts.

The lesson: build the gold set before you tune the model. Every improvement you think you're making is noise until there is a labeled set to measure against. This is the discipline that separates a demo from a product, and it is 80% of the actual engineering time.

The multilingual trap

French and Arabic in the same corpus is where a RAG system quietly falls apart, because they fail in different directions. French legal text is dense and latinate, with terms that are almost impossible to retrieve via synonyms ("nullité", "caducité", "inopposabilité" — three different legal concepts that a layperson could conflate). Arabic legal text has its own challenge: morphology. A single verb root generates a family of forms, and exact-match retrieval misses them all.

The practical answer is to stop treating language as one problem. Segment retrieval by language, embed in language-aware contexts, and — critically — let the metadata do the heavy lifting that embeddings cannot. When a question arrives in French, filter to the French corpus first; the retrieval layer never has to guess what language the source is in, because the source said so at ingest time. Multilingual retrieval is not a model problem; it is a data-structure problem dressed up as one.

The latency budget nobody budgets for

Legal users will not wait eight seconds for an answer, no matter how correct it is. The full pipeline — hybrid retrieval, reranking, generation — has to fit inside a budget where the user never notices they're waiting. That forced some unfashionable engineering: query caching for repeated questions, chunked retrieval so the reranker only ever sees a small candidate set, and a generation step that streams so the first token lands while the user is still reading the question. Latency in a professional tool is a trust signal; every wasted second reads as uncertainty.

What's next

The corpus pipeline and retrieval layer are the current focus; the answer interface is a Next.js app. When the accuracy bar is met, JurIA becomes a live product under the X-Ecosystem. If you're building legal RAG — especially for French or Arabic — I'd like to compare notes on chunking and evaluation. Get in touch.