How AI Legal Research Works Under the Hood: Retrieval, Reranking, and Why It Matters

Every legal AI research tool — Westlaw's AI-Assisted Research, Lexis+ AI, CoCounsel — does the same trick. When you ask a question, it does not answer from memory. It first searches a corpus of cases, statutes, and regulations, pulls the passages most relevant to your question, and then writes its answer using only those passages. That trick has a name: retrieval-augmented generation, or RAG. It is the single most important thing to understand about how these tools work — because the retrieval step, which you never see, decides whether the answer can be trusted at all.

This article is the under-the-hood tour: what RAG is, how the retrieval pipeline actually works (keyword search, semantic search, reranking), and why each stage matters for the quality of what lands on your desk. If you have read our earlier notes on whether you can trust an AI for legal research, this is the machinery behind those caveats.

The open-book exam

The cleanest way to think about RAG is an open-book exam. A general-purpose AI is a closed-book student: it answers from what it memorized during training, and when it does not know, it improvises. A retrieval-augmented tool is given the book — in this case, the legal corpus — and told to look things up before answering. Thomson Reuters, which builds Westlaw and CoCounsel, describes RAG exactly this way: the system "first retrieves relevant documents from a search engine and then uses those documents as inputs to the LLM in order to ground the answer," like an open-book quiz that reduces the risk of hallucination.

The technique was formalized in the 2020 paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks by Lewis and colleagues — the work that gave RAG its name. The idea is simple: instead of asking the model to recall facts, retrieve them fresh at query time and feed them in as context. For law, where the answer changes with every new opinion and every amended statute, answering from a live corpus instead of a frozen training set is not a nicety — it is the difference between current law and a snapshot from the training cutoff.

The corpus first

Before any search happens, the system needs a corpus — and in legal AI, the corpus is the moat. Westlaw's AI-Assisted Research draws on the Westlaw library with its proprietary editorial layer: the West Key Number System, headnotes written by attorney-editors, and KeyCite treatment signals. Lexis+ AI sits on the Lexis library. CoCounsel runs on the combined Westlaw and Practical Law content, the latter maintained by a team of more than 650 legal expert editors.

This matters more than it sounds. RAG is only as good as the documents it retrieves from. A general-purpose model trained on the public internet will happily retrieve from random blogs and AI-generated content farms; a legal research tool retrieves from an editorialized, jurisdiction-tagged library. That is why the same underlying model technology produces radically different answers in a legal product versus a consumer chatbot. As Thomson Reuters's lead applied scientist put it, "poor retrieval and/or bad context can be just as bad as or worse than relying on an LLM's internal memory — just as a law student using an outdated textbook will give wrong legal answers."

The retrieval step: two kinds of search

Once the corpus exists, the system has to find the relevant passages for your question. Production systems run two different searches in parallel, then combine them — a design called hybrid search.

Keyword search (BM25)

The first search is the classic information-retrieval algorithm, BM25, a refined version of the term-matching approach search engines have used for decades. It scores documents by how often your exact words appear in them, weighted by how rare those words are across the corpus. Its strength is precision on distinctive language: a statute reference, a defined term, a quotation. If you ask about "Section 2-615 of the UCC," BM25 will find the documents that literally contain those terms, even if the surrounding language is unusual.

Semantic search (vector embeddings)

The second search is the newer one. Every document is converted into a vector — a long list of numbers that captures its meaning — using a model called an embedding model. Your question is converted the same way. Documents whose vectors are close to your question's vector (measured by cosine similarity) are returned, even if they share no exact words with your query. This is what lets the system understand that "can a landlord terminate a lease if the tenant stops paying" and "eviction for nonpayment of rent" are the same question.

Neither search alone is enough. BM25 misses meaning — synonyms, paraphrases, concepts expressed differently. Semantic search misses exactness — statutory references, case names, defined terms. The research literature is consistent on this: the two methods find complementary results, and merging them recovers documents that either one alone would miss. That is why the standard production architecture runs both and fuses the results.

The reranker: quality control before the AI reads

Retrieval is built for speed, so it returns a broad candidate list — perhaps the top few hundred passages. The system then runs a reranker: a more expensive model that reads each candidate together with your question and scores how genuinely relevant it is, then reorders the list. The top handful of passages — the ones the AI will actually reason over — are the survivors.

The reranker matters more than its billing suggests. The passages that reach the AI are the only law the AI sees. If the reranker lets an irrelevant district-court opinion float above the controlling Supreme Court case, the AI will reason from the wrong authority with total confidence. In the technical literature, this two-stage design — retrieve broadly, rerank precisely — is the standard recipe: a bi-encoder retriever pulls the candidates, and a cross-encoder reranker reads each candidate against the query in detail before the final list is handed to the generator.

Why not skip retrieval and just hand the whole corpus to the model? Because context has limits. Long contexts cost more, run slower, and — per the "lost in the middle" finding from Liu and colleagues — models use information at the start and end of a long context much more reliably than information buried in the middle. Precise retrieval exists precisely because feeding a model everything guarantees it will miss the one passage that matters.

Grounding: the answer is built on the retrieved passages

With the final passages selected, the system constructs a prompt containing your question, the retrieved passages, and instructions to answer only from them — and ideally to cite them. The model's job is synthesis, not recall: read the passages, state the rule, and point at the supporting text. Westlaw's AI-Assisted Research is described this way: it "focuses the LLM on the actual language of cases, statutes, and regulations" and returns "the very best cases, statutes, and regulations" — and the best portions of them — rather than generating from the question alone.

This is the grounding that gives legal AI its credibility — and its residual risk. Grounding means the answer should be traceable: every proposition should map back to a retrieved passage. But grounding is only as good as retrieval. LegalBench-RAG, the first benchmark built specifically to evaluate the retrieval step of legal RAG systems, was created precisely because researchers found the retrieval step was being taken on faith. Built from over 79 million characters of contracts and legal documents and annotated by legal experts into 6,858 query-answer pairs, it measures whether a system finds the precise snippet that supports an answer — not just the right document, but the right sentences within it.

Where it breaks: the measured failure rates

Grounding reduces hallucination; it does not eliminate it. The Stanford study led by Magesh, Dahl, and Ho — first released in 2024 and published in the Journal of Empirical Legal Studies — tested the leading commercial legal AI tools on hundreds of open-ended queries and found incorrect or misgrounded answers in more than 17% of queries, with one major tool above 34%. The study's key distinction, which we detailed in the honest limits of AI, is between answers that are simply wrong and answers that are misgrounded — a correct rule cited to a source that does not support it.

The failures trace back to the pipeline this article describes. Retrieval misses the controlling authority; the reranker surfaces the wrong passage; the retrieved passage is stale or taken out of context; the model reads the passage but cites a different one. Each failure mode is a stage in the pipeline, which is why the fix is never "a better model" alone — it is better retrieval, better grounding, and better verification.

"The retrieval step is invisible, and it decides everything. Get the wrong passages to the AI and the AI will write you a confident, well-cited answer — to the wrong question."

What this means for how you research

The practical takeaways for a working lawyer:

• Read the cited passages, not the summary. The summary is the model's synthesis; the passages are the law. Confirm the cited text actually supports the proposition — this is the direct counter to misgrounding.
• Check that the tool searched what you need. Different tools have different corpora — federal, state, secondary, jurisdiction coverage. A tool that never retrieved the controlling state statute cannot cite it, no matter how good its model is.
• Treat retrieval as a first draft. Use the AI's authorities as a starting point, then verify history and treatment the way you always have — KeyCite, Shepard's, or your own read. The AI is faster at finding; you are still responsible for the law.
• Ask again with different words. Because retrieval is a search, rephrasing your question retrieves different passages. One query is a sample, not a census.
• Verify before reliance. ABA Formal Opinion 512 requires competence with the tool's limits (Rule 1.1), supervision of everyone who uses it (Rules 5.1 and 5.3), and protection of confidential information (Rule 1.6) — verification is the operational form of all three.

How Lawyer Assistant approaches this

Lawyer Assistant is built on the same architecture this article describes — hybrid keyword and semantic retrieval, reranking, and answers grounded strictly in retrieved passages with citations back to the source text. It runs entirely on your machine, so your queries and documents never leave your computer. The design philosophy follows from the pipeline: the tool's job is to find the right passages and cite them honestly; the lawyer's job is to verify, judge, and decide. Neither step works without the other.

The bottom line

Legal AI research is a retrieval problem wearing a language-model costume. The model that writes the answer gets the credit, but the search that finds the law — keyword and semantic retrieval, reranked and grounded — decides whether the answer is worth reading. The measured error rates and the documented failure modes all trace back to that invisible step. Use the tools for what they are: fast, broad, well-grounded researchers that still need a lawyer to read the sources, check the citations, and own the result.

Sources & further reading

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (NeurIPS 2020) — the paper that introduced RAG.
• Thomson Reuters, "Intro to Retrieval-Augmented Generation in Legal Tech" (Dec. 2024) — RAG as open-book grounding; Westlaw AI-Assisted Research mechanics; CoCounsel's Trust Team benchmarking; the >650-editor Practical Law team.
• Pipitone & Houir Alami, LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain (2024) — 6,858 expert-annotated query-answer pairs over a 79M-character legal corpus; the first benchmark for the legal retrieval step.
• Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni & Liang, Lost in the Middle: How Language Models Use Long Contexts (TACL vol. 12, 2024) — position of relevant information in long contexts affects accuracy.
• Magesh, Surani, Dahl, Suzgun, Manning & Ho, Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools (Stanford RegLab / HAI; Journal of Empirical Legal Studies, 2025) — Lexis+ AI and Ask Practical Law AI >17% incorrect or misgrounded; Westlaw AI-Assisted Research >34%.
• ABA Formal Opinion 512, "Generative Artificial Intelligence Tools" (July 29, 2024) — competence, confidentiality, candor, and supervision (full text (PDF)).

This article is general information about technology and professional practice. It is not legal advice for any specific matter, and rules vary by jurisdiction — verify against the authority applicable to your matter.