CLI Tools — Command-Line Utilities Reference

1. Quick reference

2. ingest_all.py — Bulk document ingestion

Purpose

Ingest every document in data/raw/ into ChromaDB + BM25 in one batch operation.

Usage

bashcd backend
.venv\Scripts\activate  # Windows
# source .venv/bin/activate  # macOS/Linux
python scripts/ingest_all.py

Options

Examples

Standard ingestion:

bashpython scripts/ingest_all.py

Full rebuild (wipe + re-ingest):

bashpython scripts/ingest_all.py --reset

Low-VRAM GPU (4GB):

bashpython scripts/ingest_all.py --batch-size 16

CPU-only:

bashpython scripts/ingest_all.py --device cpu --batch-size 4

What it does

1. Scans data/raw/ for supported files (.pdf, .docx, .txt, .png, .jpg, ...)
2. Parses each document with Docling (OCR for scanned PDFs)
3. Chunks via StructureAwareChunker (480 tokens, respect sections)
4. Embeds all chunks with BGE-M3 (GPU batch processing)
5. Stores vectors + metadata in ChromaDB
6. Builds BM25 sparse index from all chunks
7. Prints summary: total chunks, files processed, elapsed time

Output

Ingesting documents from data/raw/...
Found 511 files

Processing contract_A.pdf... 23 chunks
Processing agreement_B.docx... 15 chunks
...

Ingestion complete:
  Files processed: 511
  Total chunks: 18,836
  Elapsed time: 342.5s
  ChromaDB collection: legal_chunks (18,836 vectors)
  BM25 index: data/bm25_index/bm25_index.pkl (18,820 docs)

When to use

• Initial setup — first time indexing your corpus
• After adding many files — bulk re-index
• After config changes — chunk size, embedding model, etc.
• Troubleshooting — rebuild corrupted index with --reset

Notes

• Skips already-indexed files — checks ChromaDB for existing file_name (unless --reset is used)
• Backup index — saves embeddings to data/processed/all_embeddings.npy for crash recovery
• Memory usage — loads all chunks into RAM during embedding (18K chunks ≈ 2GB)

3. query.py — CLI search

Purpose

Quick CLI search for testing retrieval without the full chat pipeline.

Usage

bashcd backend
python scripts/query.py "What is rescission under contract law?"

Options

Examples

Basic search:

bashpython scripts/query.py "contract termination"

More results:

bashpython scripts/query.py "payment terms" --top-k 10

Dense-only search:

bashpython scripts/query.py "breach of contract" --search-mode dense

Skip reranking (faster):

bashpython scripts/query.py "liability" --no-rerank

Output

Query: What is rescission under contract law?
Search mode: hybrid
Reranking: enabled
Top-K: 5

Results:
1. [0.847] contract_law.pdf (p.12, §Remedies)
   "Rescission is the unmaking of a contract between parties..."

2. [0.762] remedies_guide.pdf (p.3, §Contract Remedies)
   "When a contract is rescinded, both parties are restored..."

3. [0.701] civil_code.pdf (p.145, §1689)
   "A party to a contract may rescind the contract if..."

4. [0.689] contract_basics.pdf (p.8, §Termination)
   "Rescission differs from termination in that..."

5. [0.654] legal_dictionary.pdf (p.234)
   "Rescission: noun. The cancellation of a contract..."

Latency: 4.32s

When to use

• Quick retrieval testing — verify indexed documents are searchable
• Debugging search quality — compare dense vs sparse vs hybrid
• Benchmarking retrieval speed — measure latency without LLM overhead
• Reranker A/B testing — compare --no-rerank vs default

4. chat_cli.py — Interactive chat REPL

Purpose

Interactive command-line chat interface with full RAG pipeline (retrieval + LLM answer).

Usage

bashcd backend
python scripts/chat_cli.py

Session

Legal Assistant CLI
Type 'exit' or 'quit' to end session.

> What is rescission under contract law?

[Retrieval] Searching documents...
[Retrieval] Found 5 sources in 3.2s

[LLM] Generating answer...

Rescission is the unmaking of a contract between parties, restoring them
to their pre-contract positions. Under California Civil Code §1689, a party
may rescind a contract if consent was obtained through fraud, undue influence,
or mistake. [1]

Sources:
  [1] civil_code.pdf (p.145, §1689)
  [2] contract_law.pdf (p.12, §Remedies)

> What are the payment terms?

[Retrieval] Searching documents...
[Retrieval] Found 5 sources in 2.8s

[LLM] Generating answer...

The documents do not specify payment terms. Please provide more context
about which agreement you're referring to.

> exit

Session ended. Goodbye!

Commands

When to use

• Interactive testing — chat without opening the desktop app
• Debugging LLM answers — see raw retrieval + generation
• Scripting — pipe queries for automation
• Remote access — SSH into server, run CLI chat

Notes

• Requires Ollama — LLM must be running on localhost:11434
• Uses same pipeline — identical to desktop app RAG flow
• No conversation history — each query is independent
• No approval mechanism — tools execute immediately (no HITL)

5. build_bm25_index.py — Rebuild BM25 index

Purpose

Rebuild the BM25 sparse index from existing ChromaDB chunks (useful when BM25 index is corrupted or missing).

Usage

bashcd backend
python scripts/build_bm25_index.py

What it does

1. Connects to ChromaDB (data/chroma_db)
2. Fetches all chunks from legal_chunks collection
3. Tokenizes chunk text (whitespace split)
4. Builds BM25Okapi index
5. Saves to data/bm25_index/bm25_index.pkl

Output

Building BM25 index from ChromaDB...
Fetched 18,836 chunks from legal_chunks collection
Tokenizing...
Building BM25 index...
Saved to data/bm25_index/bm25_index.pkl (18,820 documents indexed)

Index stats:
  Total documents: 18,820
  Vocabulary size: 47,234 unique tokens
  Index size: 12.3 MB

When to use

• After ChromaDB changes — BM25 out of sync with vector store
• Corrupted BM25 index — error loading .pkl file
• Missing BM25 index — data/bm25_index/bm25_index.pkl deleted

Notes

• Does NOT re-embed — uses existing ChromaDB chunks only
• Fast — no GPU needed, pure CPU tokenization (~30s for 18K chunks)
• Safe — does not modify ChromaDB

6. compare_v1_v2_retrieval.py — A/B test retrieval

Purpose

Compare two retrieval configurations side-by-side on a sample query set.

Usage

bashcd backend
python scripts/compare_v1_v2_retrieval.py

What it does

1. Loads 20 test queries
2. Runs each query through two retrieval configs:
   • V1: Dense-only, no reranking
   • V2: Hybrid (dense+sparse), reranking enabled
3. Computes Recall@5, MRR for each
4. Prints comparison table

Output

Comparing retrieval configurations on 20 queries...

Config V1: dense-only, no reranking
Config V2: hybrid, reranking enabled

Results:

| Query | V1 R@5 | V2 R@5 | V1 MRR | V2 MRR | Winner |
|-------|--------|--------|--------|--------|--------|
| Q1    | 0.8    | 1.0    | 0.6    | 1.0    | V2     |
| Q2    | 1.0    | 1.0    | 1.0    | 1.0    | Tie    |
| Q3    | 0.6    | 0.8    | 0.4    | 0.7    | V2     |
...

Summary:
  V1 wins: 3
  V2 wins: 14
  Ties: 3

  V1 Recall@5: 0.73
  V2 Recall@5: 0.86

  V1 MRR: 0.61
  V2 MRR: 0.74

When to use

• A/B testing retrieval changes — compare before/after config changes
• Evaluating reranker impact — quantify quality improvement
• Search mode comparison — dense vs sparse vs hybrid

Customization

Edit the script to test your own configs:

python# scripts/compare_v1_v2_retrieval.py
config_v1 = SearchConfig(search_mode="dense", rerank=False)
config_v2 = SearchConfig(search_mode="hybrid", rerank=True, rrf_k=60)

7. benchmark.py — Legacy benchmark runner

Purpose

Compatibility wrapper for the old benchmark interface (now deprecated; use python -m benchmark.runner instead).

Usage

bashcd backend
python scripts/benchmark.py

Equivalent to:

bashpython -m benchmark.runner

See BENCHMARKING.md for full benchmark documentation.

8. test_playbook_scan.py — Test compliance scanner

Purpose

Test the playbook compliance scanner on a single document (debugging tool).

Usage

bashcd backend
python scripts/test_playbook_scan.py <document_id> [playbook_id]

Examples

Scan with default playbook:

bashpython scripts/test_playbook_scan.py "contract_A.pdf"

Scan with specific playbook:

bashpython scripts/test_playbook_scan.py "contract_A.pdf" "gdpr"

Output

Scanning document: contract_A.pdf
Playbook: default (8 rules)

Fetching chunks... 23 chunks found
Classifying chunks...
  Chunk 1: payment_terms (confidence: 0.89)
  Chunk 2: liability_clause (confidence: 0.76)
  ...

Checking rules...
  Rule 1: "Unlimited liability clauses" → CHECKING
    Matched chunk 5 (liability_clause)
    → FLAG: Unlimited liability found (severity: HIGH)

  Rule 2: "Automatic renewal" → CHECKING
    No matches

  ...

Scan complete:
  Total chunks: 23
  Flags found: 3
    - HIGH: 1
    - MEDIUM: 2
    - LOW: 0

Flags:
  1. [HIGH] Unlimited liability clause (p.3, §5.2)
     "The licensee assumes unlimited liability for..."
     Rule: "Unlimited liability clauses"

  2. [MEDIUM] Ambiguous payment terms (p.5, §8.1)
     "Payment due within reasonable time..."
     Rule: "Vague payment obligations"

  3. [MEDIUM] Broad indemnification (p.7, §10)
     "Indemnify against all claims..."
     Rule: "Overly broad indemnification"

When to use

• Debugging scan failures — see which rules triggered
• Testing new playbooks — verify rule logic
• Inspecting clause classification — check confidence scores

9. Running scripts in production

Automated ingestion (cron)

bash# /etc/cron.daily/ingest-legal-docs
#!/bin/bash
cd /app/Lawyer-Assistant/backend
source .venv/bin/activate
python scripts/ingest_all.py >> /var/log/ingest.log 2>&1

Batch processing

bash# Process multiple queries from a file
while IFS= read -r query; do
  python scripts/query.py "$query" >> results.txt
done < queries.txt

CI/CD integration

yaml# .github/workflows/index-corpus.yml
- name: Ingest corpus
  run: |
    cd backend
    python scripts/ingest_all.py --reset
- name: Verify index
  run: |
    python scripts/query.py "test query" | grep -q "Results"

10. Common CLI workflows

Workflow 1: Add new documents and re-index

bash# 1. Copy new PDFs to data/raw/
cp ~/Downloads/*.pdf data/raw/

# 2. Re-ingest (skips already-indexed files)
cd backend
python scripts/ingest_all.py

# 3. Test a query
python scripts/query.py "new document topic"

Workflow 2: Rebuild corrupted index

bashcd backend

# Option A: Full rebuild
python scripts/ingest_all.py --reset

# Option B: Just BM25 (if ChromaDB is OK)
python scripts/build_bm25_index.py

Workflow 3: Compare retrieval configs

bash# Edit compare_v1_v2_retrieval.py with your configs
nano scripts/compare_v1_v2_retrieval.py

# Run comparison
python scripts/compare_v1_v2_retrieval.py > comparison_report.txt

# Review
cat comparison_report.txt

11. CLI script troubleshooting

12. Related documentation