Data Flow
How data actually moves through the system — from raw files to retrieval results.
1. Ingestion
Two ingestion paths exist:
- App path (workspace ingest) —
RetrievalPipeline.ingest_file(): Docling parse →StructureAwareChunker(480-token chunks, overlap 0, sections never split, perconfig.py) → embed → ChromaDB + BM25. This is what the UI uses when you attach files or run "Ingest all files". - CLI bulk path —
backend/scripts/ingest_all.py(legacy): simple char splitting (max_len=512, overlap=64). Used for one-time bulk imports only.
The diagram below shows the CLI path. Chunking happens once, in memory. Chunks are NOT saved to disk before embedding. Everything is collected in a Python list, then embedded in one batch, then saved together.
┌─────────────────────────────────────────────────────────────────────┐
│ PHASE 1: COLLECT ALL CHUNKS (in memory only, nothing on disk) │
└─────────────────────────────────────────────────────────────────────┘
data/raw/*.pdf, *.png, *.jpg data/raw/Dataset_BA.json
│ │
▼ ▼
┌──────────────────────┐ ┌─────────────────────┐
│ DocumentIngester │ │ ingest_ba_dataset() │
│ ingestion.py │ │ backend/scripts/ingest_all │
│ │ │ │
│ Docling converts │ │ Parse JSON fields: │
│ PDF/image → │ │ گروه (group) │
│ DoclingDocument │ │ عنوان (title) │
│ │ │ پیام (message) │
│ Iterates items: │ │ متن رأی (verdict) │
│ TextItem │ │ simplified text │
│ TableItem │ └──────────┬──────────┘
│ SectionHeaderItem │ │
└──────────┬───────────┘ │
│ │
▼ ▼
┌───────────────┐ ┌───────────────┐
│ Document │ │ raw text str │
│ (per block) │ │ (per record) │
│ │ │ │
│ file_name │ │ concatenation │
│ page_number │ │ of fields │
│ section_title │ └───────┬───────┘
│ content_type │ │
│ text │ │
└───────┬───────┘ │
│ │
▼ │
┌───────────────┐ │
│ chunk_text() │◄──────────────────────────┘
│ ingest_all.py │
│ │
│ Simple char │
│ splitting: │
│ max_len=512 │
│ overlap=64 │
│ │
│ No tokenizer │
│ Just string │
│ slicing │
└───────┬───────┘
│
▼
┌───────────────┐
│ Chunk │ (Pydantic model)
│ │
│ chunk_id │ sha1 hash of (file, page, section, text[:200])
│ file_name │ inherited from Document
│ page_number │ inherited from Document
│ section_title │ inherited from Document
│ text │ the actual chunk text
└───────┬───────┘
│
│ all_chunks = [] ← accumulates across ALL files
│ (in Python memory, NOT on disk)
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ PHASE 2: EMBED ALL CHUNKS (GPU, batch encoding) │
└─────────────────────────────────────────────────────────────────────┘
all_chunks (list[Chunk])
│
▼
┌───────────────┐
│ texts = │ Extract .text from each Chunk
│ [c.text for │ → list[str]
│ c in chunks] │
└───────┬───────┘
│
▼
┌───────────────────────────────────────────────┐
│ SentenceTransformer("models/bge-m3") │
│ GPU: RTX 4060 │
│ │
│ model.encode( │
│ texts, ← all chunk texts at once │
│ batch_size=32, ← configured cap; the GPU │
│ manager may tune lower │
│ normalize_embeddings=True, │
│ convert_to_numpy=True, │
│ ) │
│ │
│ Output: np.ndarray (N, 1024) float32 │
│ Each row = L2-normalized 1024-dim vector │
└───────┬───────────────────────────────────────┘
│
│ embeddings (numpy array)
│ Still in memory, NOT on disk yet
│
▼
┌───────────────────────────────────────────────┐
│ SAVE BACKUP (optional, for crash recovery) │
│ │
│ data/processed/all_embeddings.npy ← numpy │
│ data/processed/all_chunks.json ← metadata │
└───────┬───────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ PHASE 3: SAVE TO CHROMADB (persistent on disk) │
└─────────────────────────────────────────────────────────────────────┘
chunks + embeddings
│
▼
┌───────────────────────────────────────────────┐
│ VectorStore.add_chunks(chunks, embeddings) │
│ vector_store.py │
│ │
│ ChromaDB PersistentClient │
│ Path: data/chroma_db/ │
│ Collection: "legal_chunks" │
│ │
│ For each chunk: │
│ id = chunk_id │
│ embedding = 1024-dim float32 │
│ document = chunk text │
│ metadata = {file_name, page_number, │
│ section_title} │
│ │
│ Uses upsert (handles re-ingestion) │
│ Batch size: 1000 per upsert call │
│ │
│ ChromaDB handles HNSW index building │
│ Distance: cosine (= dot product for L2-norm) │
└───────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ PHASE 3.5: BUILD BM25 INDEX (separate from ChromaDB) │
└─────────────────────────────────────────────────────────────────────┘
all_chunks (same list from Phase 1)
│
▼
┌───────────────────────────────────────────────┐
│ SparseRetriever.build_index(all_chunks) │
│ sparse_retriever.py │
│ │
│ Tokenization: whitespace split (str.split()) │
│ BM25Okapi from rank_bm25 │
│ │
│ save_index() → data/bm25_index/bm25_index.pkl│
│ Contains: BM25 object + chunk_ids + texts │
└───────────────────────────────────────────────┘
FINAL STATE (on disk):
<project>/workspace/chroma_db/ ← ChromaDB (vectors + metadata + HNSW index)
<project>/workspace/bm25_index/ ← BM25 pickle (inverted index)
<project>/workspace/processed/ ← Backup (embeddings.npy + chunks.json)
When no workspace is active the paths fall back to `data/chroma_db/`,
`data/bm25_index/`, `data/processed/` — but the app requires a project folder,
so the workspace paths are the ones you'll actually see.2. Query / Retrieval
User types query
e.g. "What is rescission?"
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ RETRIEVAL PIPELINE retrieval.py → RetrievalPipeline.query() │
└─────────────────────────────────────────────────────────────────────┘
│
▼
┌───────────────┐
│ embed_query() │
│ embedder.py │
│ │
│ BGE-M3 encode │
│ (1, 1024) │
└───────┬───────┘
│ query_vec
│
┌─────┴─────┐
│ │
▼ ▼
┌────────┐ ┌────────────┐
│ DENSE │ │ SPARSE │
│ search │ │ search │
│ │ │ │
│ChromaDB│ │ BM25 │
│.query()│ │ .search() │
│ │ │ │
│ top_k= │ │ top_k= │
│ 50 │ │ 50 │
│ │ │ │
│Returns │ │ Returns │
│chunk_id│ │ chunk_id + │
│+ text │ │ BM25 score │
│+ score │ │ │
│+ meta │ │ │
└───┬────┘ └───┬────────┘
│ │
│ search_mode = "hybrid"
│ hybrid_merge_strategy = "combine_and_dedup"
│ (default; rrf / boost_only configurable)
│
▼
┌───────────────────────────────┐
│ MERGE (combine_and_dedup) │
│ │
│ Union of dense + sparse │
│ results, deduplicated by │
│ chunk id. Options: │
│ - rrf: rank fusion │
│ 1/(k + rank_dense) + │
│ 1/(k + rank_sparse) │
│ - boost_only: dense results │
│ only, BM25 boosts score │
└───────────────┬───────────────┘
│
▼
┌───────────────┐
│ RERANK │
│ reranker.py │
│ │
│ BGE-Reranker │
│ -v2-M3 │
│ │
│ Cross-encoder:│
│ [CLS] q [SEP] │
│ doc [SEP] │
│ │
│ For each of │
│ top 50 pairs: │
│ sigmoid score │
│ [0, 1] │
│ │
│ Batch: 32 │
│ Max len: 512 │
└───────┬───────┘
│
▼
┌───────────────┐
│ SORT + TOP-K │
│ │
│ Sort by │
│ reranker_score│
│ descending │
│ │
│ Filter: │
│ score >= 0.0 │
│ │
│ Take top K=5 │
└───────┬───────┘
│
▼
┌───────────────┐
│ QueryResponse │
│ │
│ .query │
│ .results[] │
│ .file_name │
│ .page │
│ .section │
│ .score │
│ .text │
│ .latency_ms │
└───────────────┘3. Key Insight: What's Saved Where
| Phase | What | Where | When |
|---|---|---|---|
| Chunking | list[Chunk] | Python memory | During Phase 1 |
| Embedding | np.ndarray (N,1024) | Python memory | During Phase 2 |
| Backup | all_embeddings.npy + all_chunks.json | data/processed/ | After Phase 2 |
| ChromaDB | vectors + text + metadata | data/chroma_db/ | Phase 3 |
| BM25 | inverted index | data/bm25_index/ | Phase 3.5 |
Chunks are NOT saved to disk independently. They exist:
- In memory during ingestion (Phase 1-2)
- As part of ChromaDB records (Phase 3)
- As part of the BM25 pickle (Phase 3.5)
- As a JSON backup (Phase 2, optional)
If ingestion crashes between Phase 2 and Phase 3, chunks are lost (the backup helps recover).
4. What Doesn't Exist Yet
- No persistent chunk cache between ingestion and embedding
- No incremental ingestion (full rebuild required)
- No chunk versioning or deduplication across runs
- No pre-computed chunk embeddings cached separately from ChromaDB