What Really Happens When You Chat With This Chatbot
What's indexed (example corpus): 511 legal documents → 18,836 chunks in ChromaDB. Your project's numbers depend on the folder you select — with a workspace active the index lives under
<project>/workspace/chroma_db.A complete, step-by-step walkthrough of every system that fires from the moment you type a message until you see the answer.
System Overview
┌─────────────┐ SSE ┌──────────────┐ HTTP ┌─────────────┐
│ Electron │ ──────────► │ Backend │ ──────────► │ Ollama │
│ Frontend │ ◄────────── │ FastAPI │ ◄────────── │ (LLM) │
│ (React/Vite)│ streaming │ (Python) │ response │ lfm2.5:8b │
└─────────────┘ └──────────────┘ └─────────────┘
│
┌──────────────┼──────────────┐
▼ ▼ ▼
┌──────────┐ ┌──────────────┐ ┌──────────┐
│ ChromaDB │ │ BGE-M3 │ │ BGE- │
│ Vector DB │ │ Embedder │ │ Reranker │
│ 18,836 │ │ (1024-dim) │ │ (cross- │
│ chunks │ │ CUDA fp16 │ │ encoder) │
└──────────┘ └──────────────┘ └──────────┘
│
▼
┌──────────┐
│ BM25 │
│ Sparse │
│ Index │
│ 18,820 │
│ chunks │
└──────────┘The Complete Flow: Step by Step
Step 0: Server Startup (happens once)
When you run npm run dev in the frontend/electron/ folder:
1. Python backend starts (FastAPI on port 8765)
2. Vite dev server starts (port 5173)
3. Electron loads the Vite URL
4. Model warmup runs in background thread:
├── BGE-M3 embedder loaded into GPU (~4s)
├── BGE-Reranker loaded into GPU (~1s)
├── Ollama LLM connection verified (~2s)
└── Classifier reference embeddings computed (~4s)
5. Health check passes: GET /api/health → 200 OKStep 1: You Type a Message
What you see: You type "hi" or "What is rescission in contract law?" and press Enter.
What happens in the frontend (App.tsx):
typescript// 1. Message is added to chat history
addMessage({ role: 'user', content: query })
// 2. A conversation ID is generated (or loaded from localStorage)
const conversationId = Date.now().toString()
// 3. POST request is sent to backend
fetch('http://localhost:8765/api/chat/stream', {
method: 'POST',
body: JSON.stringify({ query, conversation_id: conversationId })
})
// 4. The SSE stream is consumed via fetch + a line parser (streamChat in
// services/api.ts) — not EventSource, so POST body fields (mode, top_k,
// attachments, context_hint) can ride along.
Step 2: Backend Receives the Request
File: backend/main.py
python@app.post("/api/chat/stream")
async def chat_stream(request: ChatRequest):
# 1. Create an SSE streaming response
return StreamingResponse(
event_generator(request.query, request.conversation_id),
media_type="text/event-stream"
)
Step 3: Intent Classification (0.3ms)
File: backend/legal_retrieval/intent_classifier.py
The system instantly classifies your message into one of two categories:
┌─────────────────────────────────────────────────────────┐
│ INTENT CLASSIFIER │
│ │
│ Input: "hi" │
│ → Matched GREETING_EXACT set: {'hi', 'hey', ...} │
│ → Output: GREETING (0.3ms) │
│ │
│ Input: "What is rescission in contract law?" │
│ → Not a greeting pattern │
│ → Output: PLANNER (0.7ms) │
│ │
│ Input: (file attached) │
│ → File detected in message │
│ → Output: SCAN (routes to scan pipeline) │
└─────────────────────────────────────────────────────────┘Greeting detection rules:
- Exact matches:
hi,hey,hello,yo,sup,howdy,hiya,gm,ga,ge,good morning,good afternoon,good evening - Regex patterns:
^(hi|hey|hello|yo|sup|howdy|hiya)\s*[!.?]?$ - Everything else → PLANNER path
Step 4A: Greeting Path (instant, no LLM)
If intent = GREETING:
python# No LLM call, no retrieval, no tools
yield StatusEvent("✨ Generating greeting...")
yield ThinkingEvent("Greeting detected, generating friendly response")
yield TokenEvent("Hello! I'm your legal document assistant...")
yield DoneEvent(answer="Hello! ...", sources=[], ...)
Total time: <100ms (string formatting only)
Step 4B: Planner Path (the main flow)
File: backend/legal_retrieval/router_intent.py
Step 4B.1: Planner Node (LLM call #1 — plan only)
┌─────────────────────────────────────────────────────────┐
│ PLANNER NODE │
│ │
│ Model: lfm2.5:8b via Ollama │
│ │
│ System Prompt: │
│ "You are a legal research planner. Your ONLY job is │
│ to write a brief plan and output a DIRECTIVE. │
│ You NEVER answer the question yourself." │
│ │
│ DIRECTIVES available: │
│ SEARCH: <query> — search indexed documents │
│ SCAN: <document_id> — scan document against playbook │
│ INGEST: <file_path> — ingest a new document │
│ ANSWER: — answer directly (no tools) │
│ │
│ User: "What is rescission in contract law?" │
│ │
│ LLM Output: │
│ "I will search for documents about rescission in │
│ contract law." │
│ DIRECTIVE: SEARCH: What is rescission in contract law? │
│ │
│ Tokens: ~800 generated in ~2-3s │
│ Status events streamed to frontend as thinking tokens │
└─────────────────────────────────────────────────────────┘Step 4B.2: Directive Parser
The directive text is parsed to extract the tool to call and its arguments:
python# Parser extracts:
tool_name = "SEARCH"
tool_input = "What is rescission in contract law?"
# The exact user query is preserved — not modified by the LLM
Step 5: Tool Execution — search_documents
File: backend/legal_retrieval/tools.py → _search_documents_impl()
This is where the heavy lifting happens. The search has 6 sub-steps:
Step 5.1: Query Embedding (BGE-M3)
┌─────────────────────────────────────────────────────────┐
│ BGE-M3 EMBEDDER (CUDA) │
│ │
│ Input: "What is rescission in contract law?" │
│ Model: BAAI/bge-m3 (1024-dim, XLM-RoBERTa) │
│ Device: CUDA (GPU) │
│ Normalization: L2 (for cosine similarity) │
│ │
│ Output: [0.0234, -0.0156, 0.0891, ...] (1024 floats) │
│ │
│ Time: ~5s first query (model load) │
│ ~0.1s subsequent queries (cached) │
└─────────────────────────────────────────────────────────┘Step 5.2: Dense Retrieval (ChromaDB)
┌─────────────────────────────────────────────────────────┐
│ CHROMADB DENSE SEARCH │
│ │
│ Collection: legal_chunks (18,836 vectors) │
│ Metric: cosine similarity (HNSW index) │
│ top_k: 50 │
│ │
│ Operation: Find 50 chunks most similar to query vector │
│ Result: 50 chunks with cosine scores (0.0 - 1.0) │
│ │
│ Time: ~0.5s │
└─────────────────────────────────────────────────────────┘Step 5.3: Sparse Retrieval (BM25)
┌─────────────────────────────────────────────────────────┐
│ BM25 SPARSE SEARCH │
│ │
│ Index: 18,820 chunks (loaded from pickle) │
│ Tokenizer: whitespace splitting │
│ top_k: 50 │
│ │
│ Operation: Keyword matching — finds chunks containing │
│ "rescission", "contract", "law" even if embedding │
│ similarity is low │
│ │
│ Result: 50 chunks with BM25 scores │
│ │
│ Time: ~0.3s │
└─────────────────────────────────────────────────────────┘Why both? Dense catches semantic meaning ("contract termination" ≈ "rescission"). Sparse catches exact keywords ("Rescission" appears verbatim in a clause). Neither alone is perfect.
Step 5.4: Hybrid Merge (combine_and_dedup)
The default merge strategy is combine_and_dedup (union of dense + sparse results, deduplicated by chunk id); rrf (reciprocal rank fusion) and boost_only are configurable alternatives.
┌─────────────────────────────────────────────────────────┐
│ HYBRID MERGE (combine_and_dedup) │
│ │
│ Input: │
│ 50 dense results (by embedding similarity) │
│ 50 sparse results (by BM25 keyword match) │
│ │
│ Process: │
│ 1. Collect all unique chunk_ids from both lists │
│ 2. Deduplicate by chunk_id │
│ 3. For each unique chunk: │
│ - If in BOTH: mark as "hybrid" (strongest signal) │
│ - If dense-only: mark as "dense" │
│ - If sparse-only: mark as "sparse" │
│ │
│ Output: ~80-100 unique chunks │
│ │
│ Time: <10ms (set operations) │
└─────────────────────────────────────────────────────────┘Step 5.5: Reranking (BGE-Reranker-v2-M3)
┌─────────────────────────────────────────────────────────┐
│ CROSS-ENCODER RERANKER │
│ │
│ Model: BAAI/bge-reranker-v2-m3 │
│ Device: CUDA (GPU), fp16, batch_size=32 │
│ │
│ Input: 100 (query, chunk) pairs │
│ │
│ For each pair, the cross-encoder computes: │
│ P(relevant | query, chunk) → score in [0, 1] │
│ │
│ Unlike the embedder (which encodes query and doc │
│ SEPARATELY), the reranker processes them TOGETHER │
│ in a single forward pass — much more accurate but │
│ much slower. │
│ │
│ Process: │
│ Batch 1: pairs 1-32 → scores │
│ Batch 2: pairs 33-64 → scores │
│ Batch 3: pairs 65-96 → scores │
│ Batch 4: pairs 97-100 → scores │
│ │
│ Output: 100 reranker_scores (one per chunk) │
│ │
│ Time: ~8s (with fp16 + batch_size=32) │
│ ~300s without optimization │
└─────────────────────────────────────────────────────────┘Step 5.6: Final Selection
┌─────────────────────────────────────────────────────────┐
│ FINAL SELECTION │
│ │
│ 1. Sort 100 chunks by reranker_score descending │
│ 2. Dedup by file_name (keep highest-scoring per file) │
│ → prevents one contract from filling all 5 slots │
│ 3. Take top 5 (top_k_final) │
│ │
│ Each result contains: │
│ - file_name: "Arconic_Trademark_License.pdf" │
│ - page: 3 │
│ - section: "License Grant" │
│ - score: 0.82 (reranker confidence) │
│ - text: chunk text (~480 tokens) │
│ - source_text: full original paragraph │
└─────────────────────────────────────────────────────────┘Step 6: Answer Generation (LLM call #2)
File: backend/legal_retrieval/router_intent.py — answer node
┌─────────────────────────────────────────────────────────┐
│ ANSWER NODE │
│ │
│ Model: lfm2.5:8b via Ollama │
│ │
│ System Prompt: │
│ "You are a legal document answerer. Base your answer │
│ on the document text below. You MAY paraphrase and │
│ synthesize. You MAY draw reasonable inferences. │
│ NEVER invent facts. Cite file name and page." │
│ │
│ Context (6000 chars max): │
│ 1. [0.82] Arconic_Trademark_License.pdf, page 3 │
│ "The License Grant hereby grants..." │
│ 2. [0.71] Arconic_Trademark_License.pdf, page 5 │
│ "Territory: United States and Canada..." │
│ 3. [0.65] Other_Document.pdf, page 12 │
│ ... │
│ │
│ User: "What is rescission in contract law?" │
│ │
│ LLM Output (streamed token by token): │
│ "Rescission is the unmaking of a contract between │
│ parties. Under [Arconic_Trademark_License.pdf, p.3], │
│ the agreement may be rescinded if either party..." │
│ │
│ Tokens: ~500-1000 generated in ~5-15s │
│ Each token is sent as a TokenEvent via SSE │
└─────────────────────────────────────────────────────────┘Step 7: Response Streaming to Frontend
File: backend/main.py — SSE event generator
┌─────────────────────────────────────────────────────────┐
│ SSE EVENT STREAM │
│ │
│ event: status │
│ data: {"type":"status","message":"🔍 Searching..."} │
│ │
│ event: thinking │
│ data: {"type":"thinking","message":"Planning search"} │
│ │
│ event: token │
│ data: {"type":"token","content":"Rescission"} │
│ │
│ event: token │
│ data: {"type":"token","content":" is"} │
│ │
│ event: token │
│ data: {"type":"token","content":" the"} │
│ │
│ ... (hundreds of token events) ... │
│ │
│ event: tool_call │
│ data: {"type":"tool_call","tool":"search_documents"} │
│ │
│ event: tool_result │
│ data: {"type":"tool_result","tool":"search_documents", │
│ "result":"Found 5 documents..."} │
│ │
│ event: done │
│ data: {"type":"done","answer":"Rescission is...", │
│ "sources":[{file_name, page, section, score, │
│ text, source_text}], │
│ "metrics":{...}} │
│ │
│ event: error (only if something fails) │
│ data: {"type":"error","message":"..."} │
└─────────────────────────────────────────────────────────┘Step 8: Frontend Renders the Response
File: frontend/src/components/Chat/Message.tsx
┌─────────────────────────────────────────────────────────┐
│ FRONTEND RENDERING │
│ │
│ 1. User message bubble appears instantly │
│ │
│ 2. Typing indicator shows "Thinking..." │
│ │
│ 3. Thinking tokens appear in collapsible "Process" │
│ section (shows planner's plan) │
│ │
│ 4. Tool call/status events show pipeline steps: │
│ ├── "🔍 Searching documents..." │
│ ├── "📊 Embedding query..." │
│ ├── "🔗 Merging dense + sparse..." │
│ └── "🎯 Reranking candidates..." │
│ │
│ 5. Answer tokens stream in word by word │
│ (Message component appends each token) │
│ │
│ 6. Source citations appear as numbered badges [1] [2] │
│ in the Sources panel on the right │
│ │
│ 7. DoneEvent triggers: │
│ ├── Final answer rendered as Markdown │
│ ├── Sources panel populated with clickable cards │
│ └── Clicking a source opens SourceViewerModal │
│ showing the FULL original paragraph │
└─────────────────────────────────────────────────────────┘Complete Timeline for a Legal Query
0ms ── POST /api/chat/stream received
0.3ms ── Intent classified: PLANNER
50ms ── Planner node: LLM call starts (plan generation)
2500ms ── Planner output: "SEARCH: What is rescission..."
2600ms ── search_documents tool invoked
2700ms ── BGE-M3: query embedded (~100ms if cached)
3200ms ── ChromaDB: 50 dense results (~500ms)
3500ms ── BM25: 50 sparse results (~300ms)
3510ms ── Merge: 100 unique chunks (<10ms)
3520ms ── BGE-Reranker: starts processing 100 pairs
11500ms ── Reranker: done (~8s with fp16+batch32)
11510ms ── Top 5 selected, deduped by file
11600ms ── Answer node: LLM call starts (answer generation)
11600ms ── First tokens streamed to frontend
16600ms ── Last token streamed
16700ms ── DoneEvent sent with sources and metrics
16800ms ── Frontend: answer fully rendered
Total: ~17 seconds (first query, model warmup included)
~12 seconds (subsequent queries, models cached)Breakdown:
| Phase | Time | % of Total |
|---|---|---|
| Intent classification | <1ms | 0% |
| Planner (LLM #1) | ~2.5s | 15% |
| Query embedding | ~0.1s | 1% |
| Dense retrieval | ~0.5s | 3% |
| Sparse retrieval | ~0.3s | 2% |
| Merge | <0.01s | 0% |
| Reranking | ~8s | 47% |
| Answer generation (LLM #2) | ~5s | 30% |
| Network overhead | ~0.5s | 3% |
The reranker is the bottleneck — nearly half the total time.
The Scan Path (when you attach a file)
When you attach a document, the intent classifier routes to the scan pipeline instead:
┌─────────────────────────────────────────────────────────┐
│ SCAN PIPELINE │
│ │
│ 1. File is saved into the active project folder │
│ 2. Docling extracts blocks (paragraphs, headings, etc) │
│ 3. StructureAwareChunker splits into ~480-token chunks │
│ 4. BGE-M3 embeds all chunks (GPU batch) │
│ 5. Chunks stored in ChromaDB (with source_text) │
│ 6. BM25 index updated │
│ 7. Playbook scanner runs: │
│ ├── Classify each chunk's clause type │
│ ├── Match to playbook rules (8 rules in default) │
│ ├── LLM checks compliance per (chunk, rule) pair │
│ └── Generate compliance flags sorted by severity │
│ 8. Flags displayed as structured cards in chat │
│ │
│ Total for a 3-page doc: ~30-140s │
│ (depends on number of chunks × rules checked) │
└─────────────────────────────────────────────────────────┘Models Used
| Model | Purpose | Size | Device | Latency |
|---|---|---|---|---|
| lfm2.5:8b | LLM (planner + answer) | 8B params | Ollama | 2-15s per call |
| BGE-M3 | Query/chunk embedding | 568M params | CUDA fp16 | 0.1s cached, 4s cold |
| BGE-Reranker-v2-M3 | Cross-encoder reranking | 568M params | CUDA fp16 | 8s for 100 pairs |
| BM25 | Keyword search | N/A (index) | CPU | 0.3s |
Data Storage
| Store | Contents | Location |
|---|---|---|
| ChromaDB | chunk vectors + metadata | <project>/workspace/chroma_db/ (workspace active) or data/chroma_db/ |
| BM25 Index | Keyword search index (pickle) | <project>/workspace/bm25_index/ (workspace active) or data/bm25_index/ |
| SQLite | Conversation memory (checkpointer) | data/conversations.db (project-independent) |
| SQLite | Scan compliance flags | data/flags.db (project-independent) |
| Filesystem | Source documents | the active project folder itself (workspace required — no default data/raw/ anymore) |
| Filesystem | ML model weights | models/ |
Key Design Decisions
- Two LLM calls per query — planner decides WHAT to search, answer writes HOW to respond. Separation of concerns.
- Hybrid search (dense + sparse) — dense catches meaning, sparse catches exact terms. Neither alone is sufficient for legal text.
- Cross-encoder reranking — much more accurate than embedding similarity alone, but 50x slower. Worth it for accuracy.
- Anti-hallucination prompt — the answer node is told to NEVER invent facts, only use document text with citations.
- source_text storage — full original paragraphs stored alongside chunks so the SourceViewerModal shows complete context.
- Greeting bypass — "hi" gets an instant response without touching the LLM or retrieval pipeline.
- HITL (Human-in-the-loop) — the agent can interrupt before taking actions, waiting for user approval via the resume endpoint.