Benchmarking — Performance Evaluation & Quality Metrics

1. Overview

The benchmarking system evaluates three dimensions:

1. Retrieval quality — How well does the system find relevant documents?
2. Component performance — How fast are individual components (chunking, embedding, reranking)?
3. End-to-end accuracy — How accurate are the final answers?

Benchmark suite structure

backend/benchmark/
├── runner.py                  # Main benchmark orchestrator
├── metrics.py                 # Retrieval metrics (Recall@K, MRR, Precision@K)
├── failure_analysis.py        # Failure categorization & diagnosis
├── connector.py               # Pipeline integration
├── datasets/                  # Benchmark datasets
│   ├── benchmark_dataset.py   # Main question set (200 questions)
│   ├── golden_eval.py         # Golden evaluation set
│   ├── golden_eval_v2.py      # V2 with extended metadata
│   └── cuad_dataset.py        # CUAD contract understanding
├── components/                # Component-level benchmarks
│   ├── test_chunking.py       # Chunking speed & quality
│   ├── test_embedding.py      # Embedding throughput
│   ├── test_reranker.py       # Reranking speed vs quality
│   └── test_retrieval.py      # Retrieval accuracy
└── results/                   # Benchmark outputs
    ├── results.json           # Per-question results
    ├── metrics.json           # Aggregate metrics
    ├── failures.json          # Failure analysis
    ├── benchmark_report.md    # Human-readable report
    └── golden_eval_metrics.json   # Golden eval results

2. Running benchmarks

Full retrieval benchmark (200 questions)

bashcd backend
python -m benchmark.runner

What it does:

• Loads 200 legal questions with expected sources
• Runs each query through the retrieval pipeline
• Computes Recall@K, MRR, Precision@K, keyword match
• Generates failure analysis
• Produces benchmark_report.md

Expected runtime: ~15-25 minutes (depends on GPU)

Subset benchmark (for quick iteration)

bashcd backend
python -m benchmark.runner --max 20

Runs only the first 20 questions (~2 minutes).

Benchmark with different configurations

bash# Disable reranker (tests dense+sparse only)
python -m benchmark.runner --skip-reranker

# Different top-K
python -m benchmark.runner --top-k 20

# Combine options
python -m benchmark.runner --max 50 --top-k 15 --skip-reranker

3. Benchmark metrics

Recall@K

Definition: Fraction of queries where at least one expected source appears in the top-K results.

Recall@K = (# queries with ≥1 expected source in top-K) / (total queries)

Example:

• Query: "What is rescission?"
• Expected: ["contract_law.pdf"]
• Top-5 results: ["other.pdf", "contract_law.pdf", "terms.pdf", ...]
• Recall@5 = 1.0 (expected source found)

Interpretation:

• Recall@1 = 0.75 → 75% of queries have the right document as #1 result
• Recall@5 = 0.92 → 92% of queries have the right document in top-5
• Recall@10 = 0.95 → 95% of queries have the right document in top-10

Mean Reciprocal Rank (MRR)

Definition: Average of 1 / rank where rank is the position of the first expected source.

MRR = mean(1 / rank_of_first_relevant)

Example:

• Query 1: Expected source is #1 → 1/1 = 1.0
• Query 2: Expected source is #3 → 1/3 = 0.333
• Query 3: Expected source not in top-10 → 0.0
• MRR = (1.0 + 0.333 + 0.0) / 3 = 0.444

Interpretation:

• MRR = 1.0 → perfect (every query's top result is correct)
• MRR = 0.5 → expected source typically at rank 2
• MRR = 0.25 → expected source typically at rank 4

Precision@K

Definition: Fraction of top-K results that are relevant (in expected sources).

Precision@K = (# relevant docs in top-K) / K

Example:

• Expected: ["contract.pdf", "terms.pdf"]
• Top-5: ["contract.pdf", "other.pdf", "terms.pdf", "misc.pdf", "random.pdf"]
• Precision@5 = 2/5 = 0.4

Interpretation:

• Precision@3 = 0.8 → 80% of top-3 results are relevant
• Precision@5 = 0.6 → 60% of top-5 results are relevant

Keyword match rate

Definition: Fraction of expected keywords found in the text of the top result.

KW Match = (# expected keywords in top result) / (total expected keywords)

Example:

• Expected keywords: ["rescission", "contract", "termination"]
• Top result text: "The contract may be rescinded..."
• Found: 2/3 ("rescission", "contract") → 0.667

Interpretation:

• KW match = 1.0 → top result contains all expected keywords
• KW match = 0.5 → top result contains half the expected keywords
• KW match = 0.0 → top result contains none (likely wrong document)

4. Benchmark dataset

The main benchmark uses 200 hand-curated legal questions with ground truth:

python# backend/benchmark/datasets/benchmark_dataset.py
QUESTIONS = [
    {
        "id": "Q001",
        "question": "What is rescission under contract law?",
        "category": "contract_law",
        "difficulty": "easy",
        "expected_sources": ["contract_basics.pdf", "remedies.pdf"],
        "expected_keywords": ["rescission", "contract", "void", "remedy"],
    },
    # ... 199 more
]

Dataset statistics

Get stats programmatically:

pythonfrom benchmark.datasets.benchmark_dataset import get_dataset_stats
stats = get_dataset_stats()
# → {"total": 200, "categories": {...}, "difficulties": {...}}

5. Reading benchmark results

After running python -m benchmark.runner, results are saved to backend/benchmark/results/:

benchmark_report.md (human-readable)

markdown# Legal RAG Benchmark Report

**Date**: 2026-08-03 14:23:45
**Top-K**: 10
**Reranker**: Enabled
**Total time**: 892.3s

## Summary

| Metric | Value |
|--------|-------|
| Total questions | 200 |
| Success rate | 87.5% |
| Recall@1 | 0.695 |
| Recall@5 | 0.850 |
| Recall@10 | 0.875 |
| MRR | 0.782 |
| Precision@3 | 0.623 |
| Precision@5 | 0.534 |
| Keyword match rate | 0.712 |
| Mean latency | 4461ms |

## Per-Category Breakdown

| Category | N | Success | R@1 | R@5 | MRR | KW Match |
|----------|---|---------|-----|-----|-----|----------|
| contract_law | 60 | 92% | 0.733 | 0.900 | 0.810 | 0.756 |
| tort_law | 40 | 85% | 0.675 | 0.825 | 0.748 | 0.682 |
...

metrics.json (machine-readable)

json{
  "total_questions": 200,
  "successful": 175,
  "failed": 25,
  "success_rate": 0.875,
  "recall_at_1": 0.695,
  "recall_at_5": 0.850,
  "recall_at_10": 0.875,
  "mrr": 0.782,
  "precision_at_3": 0.623,
  "precision_at_5": 0.534,
  "keyword_match_rate": 0.712,
  "mean_latency_ms": 4461.2,
  "per_category": { ... },
  "per_difficulty": { ... },
  "total_time_s": 892.3,
  "reranker_enabled": true
}

results.json (per-question details)

json[
  {
    "question_id": "Q001",
    "question": "What is rescission under contract law?",
    "category": "contract_law",
    "difficulty": "easy",
    "expected_sources": ["contract_basics.pdf"],
    "expected_keywords": ["rescission", "contract"],
    "is_success": true,
    "recall_at_1": 1.0,
    "recall_at_5": 1.0,
    "recall_at_10": 1.0,
    "mrr": 1.0,
    "precision_at_3": 0.667,
    "precision_at_5": 0.400,
    "keyword_match_rate": 1.0,
    "latency_ms": 4523.1,
    "failure_type": "",
    "retrieved_docs": [
      {"document_id": "contract_basics.pdf", "score": 0.89},
      {"document_id": "remedies.pdf", "score": 0.76},
      ...
    ]
  },
  ...
]

failures.json (failure analysis)

json{
  "total_questions": 200,
  "total_failed": 25,
  "failure_rate": 0.125,
  "failure_type_counts": {
    "missing_data": 10,
    "embedding_problem": 8,
    "ocr_problem": 4,
    "reranker_problem": 3
  },
  "worst_failures": [
    {
      "question_id": "Q087",
      "question": "What is the doctrine of adverse possession?",
      "category": "property_law",
      "difficulty": "hard",
      "failure_type": "missing_data",
      "mrr": 0.0,
      "possible_fix": "Add more property law documents to corpus"
    },
    ...
  ]
}

6. Failure analysis

The benchmark automatically categorizes failures:

Diagnosing a failure

bashcd backend
python -m benchmark.failure_analysis

Opens an interactive session:

python>>> from benchmark.runner import run_benchmark
>>> results = run_benchmark(max_questions=200)
>>> from benchmark.failure_analysis import analyze_failures
>>> analysis = analyze_failures(results['results'])
>>> print(analysis['worst_failures'][0])
{
  "question_id": "Q087",
  "question": "What is the doctrine of adverse possession?",
  "category": "property_law",
  "failure_type": "missing_data",
  "possible_fix": "Add more property law documents to corpus"
}

7. Golden evaluation

The golden eval is a smaller, curated set of high-confidence questions with verified answers — used to track answer quality (not just retrieval).

Running golden eval

bashcd backend
python -m benchmark.run_golden_eval

What it does:

• Loads 50 golden questions with verified correct answers
• Runs each through the full RAG pipeline (retrieval + LLM answer)
• Compares generated answer to expected answer (lexical + semantic similarity)
• Computes answer accuracy metrics

Golden eval metrics

json{
  "total": 50,
  "exact_match": 0.24,
  "f1_score": 0.78,
  "semantic_similarity": 0.82,
  "citation_accuracy": 0.91,
  "abstain_rate": 0.06
}

8. Component benchmarks

Test individual components in isolation:

Chunking performance

bashcd backend
python -m benchmark.components.test_chunking

Tests:

• Chunks per second (throughput)
• Chunk size distribution (are chunks near target 480 tokens?)
• Section respect (chunks never split headers)

Output:

Chunking performance:
  Documents processed: 100
  Total chunks: 2,340
  Throughput: 156.7 chunks/sec
  Mean chunk size: 478.2 tokens (target: 480)
  Section violations: 0 (0.0%)

Embedding performance

bashcd backend
python -m benchmark.components.test_embedding

Tests:

• Embedding throughput (chunks/sec)
• GPU vs CPU speed
• Batch size impact

Output:

Embedding performance (GPU, batch=32):
  Chunks embedded: 1,000
  Throughput: 89.3 chunks/sec
  Mean latency per chunk: 11.2ms
  GPU memory peak: 2,341 MB

Reranker performance

bashcd backend
python -m benchmark.components.test_reranker

Tests:

• Reranking throughput (pairs/sec)
• Quality impact (how much does reranking improve Recall@5?)
• Speed vs batch size trade-off

Output:

Reranker performance:
  Query-doc pairs reranked: 500
  Throughput: 18.7 pairs/sec
  Recall@5 improvement: +12.3%
  Mean score change: +0.18 (top result promoted)

Retrieval accuracy

bashcd backend
python -m benchmark.components.test_retrieval

Tests:

• Dense vs sparse vs hybrid accuracy
• Top-K impact on recall
• Search mode comparison

Output:

Retrieval mode comparison (100 queries):

Dense only:
  Recall@5: 0.73
  MRR: 0.61

Sparse only:
  Recall@5: 0.68
  MRR: 0.54

Hybrid (dense + sparse):
  Recall@5: 0.86
  MRR: 0.74

9. Continuous benchmarking

Tracking performance over time

Save metrics after each benchmark run:

bashcd backend
python -m benchmark.runner > results/run_$(date +%Y%m%d_%H%M%S).log
cp results/metrics.json results/metrics_$(date +%Y%m%d_%H%M%S).json

Plot trends:

pythonimport json
from pathlib import Path

metrics_files = sorted(Path("results").glob("metrics_*.json"))
recalls = []
for f in metrics_files:
    data = json.loads(f.read_text())
    recalls.append(data["recall_at_5"])

print(f"Recall@5 over {len(recalls)} runs: {recalls}")
# → [0.82, 0.84, 0.86, 0.87, 0.85, ...]

Regression detection

Set a minimum acceptable threshold:

pythonimport json

with open("results/metrics.json") as f:
    metrics = json.load(f)

THRESHOLDS = {
    "recall_at_5": 0.80,
    "mrr": 0.70,
    "success_rate": 0.75,
}

for metric, min_val in THRESHOLDS.items():
    actual = metrics[metric]
    if actual < min_val:
        raise AssertionError(
            f"{metric} regression: {actual:.3f} < {min_val:.3f}"
        )

Run in CI:

yaml# .github/workflows/benchmark.yml
- name: Run benchmark
  run: cd backend && python -m benchmark.runner --max 50
- name: Check regressions
  run: python scripts/check_benchmark_regression.py

10. Benchmark configuration

All benchmark config is in backend/benchmark/connector.py:

pythondef create_pipeline():
    """Create retrieval pipeline for benchmarking."""
    return RetrievalPipeline(
        top_k_retrieval=50,
        top_k_final=10,
        search_mode="hybrid",
        skip_rerank=False,
    )

Override for custom configs:

pythonfrom benchmark.connector import create_pipeline
from benchmark.runner import run_benchmark

# Custom pipeline
pipeline = create_pipeline()
pipeline.top_k_final = 20
pipeline.skip_rerank = True

# Run with custom pipeline
results = run_benchmark(max_questions=100, top_k=20)

11. Adding new benchmark questions

Edit backend/benchmark/datasets/benchmark_dataset.py:

pythonQUESTIONS = [
    {
        "id": "Q201",
        "question": "What is the statute of limitations for breach of contract?",
        "category": "contract_law",
        "difficulty": "medium",
        "expected_sources": ["contract_remedies.pdf", "limitations_act.pdf"],
        "expected_keywords": ["statute", "limitations", "breach", "contract"],
    },
    # ... add more
]

Question guidelines

Good question:

• Specific, answerable from corpus
• Expected sources exist in data/raw/
• Keywords are distinctive (not generic stopwords)
• Difficulty matches query complexity

Bad question:

• Too broad ("Tell me about law")
• Expected source not in corpus
• Keywords too generic ("law", "legal", "document")

12. Benchmarking best practices

Before benchmarking

1. ✅ Ingest corpus — ensure all documents are indexed
2. ✅ Restart backend — cold start (models warm up on first query)
3. ✅ Close other apps — free GPU memory
4. ✅ Use consistent config — same top_k, skip_rerank, etc.

Interpreting results

Good retrieval:

• Recall@5 ≥ 0.85
• MRR ≥ 0.70
• Keyword match ≥ 0.65

Good performance:

• Mean latency ≤ 5 seconds per query
• Success rate ≥ 80%

Red flags:

• Recall@10 < 0.70 → corpus missing key documents
• MRR < 0.50 → retrieval returning irrelevant results
• Keyword match < 0.50 → wrong documents retrieved

Improving metrics

13. Benchmark CI integration

Run benchmarks in CI on every PR:

yaml# .github/workflows/benchmark.yml
name: Benchmark
on: [pull_request]
jobs:
  benchmark:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - name: Install dependencies
        run: |
          cd backend
          pip install -r requirements.txt
      - name: Run benchmark subset
        run: |
          cd backend
          python -m benchmark.runner --max 20
      - name: Check regressions
        run: |
          python scripts/check_benchmark_thresholds.py
      - name: Upload results
        uses: actions/upload-artifact@v3
        with:
          name: benchmark-results
          path: backend/benchmark/results/

14. CUAD benchmark

The CUAD (Contract Understanding Atticus Dataset) benchmark tests contract understanding:

bashcd backend
python -m benchmark.cuad_runner

What it tests:

• Clause extraction from contracts
• Contractual term identification
• Legal concept recognition

Metrics:

• Exact match accuracy
• F1 score (token-level overlap)
• Clause recall (fraction of clauses found)

See backend/benchmark/cuad_runner.py for details.

15. Troubleshooting benchmarks

"Pipeline not initialized"

Cause: ChromaDB not populated

Fix:

bashcd backend
python scripts/ingest_all.py

Benchmark very slow

Cause: Running on CPU, not GPU

Fix: Check GPU availability:

pythonimport torch
print(torch.cuda.is_available())  # should be True

Set device explicitly:

bashexport PLR_DEVICE=cuda
python -m benchmark.runner

OOM during benchmark

Cause: Batch size too large

Fix: Lower batch size:

bashexport PLR_EMBEDDING_BATCH_SIZE=16
export PLR_RERANKER_BATCH_SIZE=16
python -m benchmark.runner

Results don't match README

Cause: Different corpus or config

Fix: Use the exact corpus + config from the README:

• Corpus: 511 legal documents (18,836 chunks)
• Config: hybrid search, reranker enabled, top_k=5

16. Benchmark visualization

Generate charts from results

After running a benchmark, generate visual charts:

bashcd backend
python -m benchmark.visualization

Generated charts (saved to backend/benchmark/results/charts/):

1. recall_comparison.png - Bar chart comparing Recall@1, R@5, R@10
2. category_breakdown.png - Per-category Recall@5 horizontal bars
3. metrics_overview.png - 4-panel dashboard with all key metrics
4. latency_distribution.png - Histogram of query latencies
5. failure_analysis.png - Pie chart of failure types
6. performance_over_time.png - Line chart tracking metrics across runs

Installation

Visualization requires matplotlib:

bashcd backend
pip install matplotlib

Custom input

Specify a different metrics file:

bashpython -m benchmark.visualization --input path/to/metrics.json

Performance tracking workflow

Track improvements across development:

bash# Run benchmark and save timestamped results
cd backend
python -m benchmark.runner
cp results/metrics.json results/metrics_$(date +%Y%m%d_%H%M%S).json

# After multiple runs, generate time-series chart
python -m benchmark.visualization
# → Creates performance_over_time.png showing trends

Example charts

Recall Comparison:

• Bar chart: R@1, R@5, R@10
• Quickly see retrieval quality at different K values

Category Breakdown:

• Horizontal bars: Recall@5 per category
• Identify which legal domains perform best/worst

Metrics Overview:

• 4-panel dashboard: Recall, Precision, Success by difficulty, Summary stats
• One-page snapshot of entire benchmark

Latency Distribution:

• Histogram: Query latency distribution
• Mean/median lines for reference
• Identify slow outliers

Failure Analysis:

• Pie chart: Breakdown of failure types
• Prioritize what to fix next

Performance Over Time:

• Line chart: R@1, R@5, MRR across multiple runs
• Track improvement/regression trends

Exporting for reports

Charts are saved as high-DPI PNG (150 DPI) for inclusion in:

• Technical documentation
• Progress reports
• Stakeholder presentations
• GitHub README

Programmatic usage

pythonfrom benchmark.visualization import generate_all_visualizations
from pathlib import Path

# Generate charts
generate_all_visualizations(Path("results/metrics.json"))

# Or individual charts
from benchmark.visualization import plot_recall_comparison, load_results

metrics = load_results(Path("results/metrics.json"))
plot_recall_comparison(metrics, Path("output/recall.png"))

17. Related documentation