Storage — Database Schemas, Data Models & Persistence

1. Storage overview

Storage locations

Workspace vs global storage

Workspace-scoped (per-project):

• Vector store (ChromaDB)
• BM25 index
• Chat history + working set + GPU profile
• Raw uploads

Global (shared across projects):

• LangGraph conversation threads
• Compliance flags
• Models
• Logs

2. ChromaDB schema (vector store)

Collection structure

Collection name: legal_chunks

Distance metric: Cosine (dot product for L2-normalized vectors)

Index: HNSW (Hierarchical Navigable Small World)

Document schema

Each document in ChromaDB represents one chunk:

python{
  "id": "abc123...def",              # chunk_id (SHA1 hash)
  "embedding": [0.023, -0.045, ...], # 1024-dim float32 vector (L2-normalized)
  "document": "The contract may be rescinded...",  # chunk text
  "metadata": {
    "file_name": "contract_law.pdf",
    "page_number": 12,
    "section_title": "Remedies",
    "source_text": "The contract may be rescinded under the following conditions: ..."
  }
}

Metadata fields

Chunk ID generation

pythonimport hashlib

def generate_chunk_id(file_name: str, page: int, section: str, text: str) -> str:
    """Stable, collision-free chunk ID."""
    key = f"{file_name}|{page}|{section}|{text[:200]}"
    return hashlib.sha1(key.encode()).hexdigest()

Properties:

• Stable (same input → same ID)
• Collision-resistant (SHA1)
• Unique per (file, page, section, text prefix)

Collection metadata

pythoncollection.metadata = {
    "hnsw:space": "cosine",
    "hnsw:construction_ef": 100,
    "hnsw:M": 16,
    "hnsw:search_ef": 128,
}

Querying ChromaDB

pythonfrom chromadb import PersistentClient

client = PersistentClient(path="data/chroma_db")
collection = client.get_collection("legal_chunks")

# Query by vector
results = collection.query(
    query_embeddings=[[0.023, -0.045, ...]],  # 1024-dim
    n_results=50,
    include=["documents", "metadatas", "distances"]
)

# Query by text (auto-embedding)
results = collection.query(
    query_texts=["What is rescission?"],
    n_results=50
)

# Filter by metadata
results = collection.query(
    query_texts=["payment terms"],
    where={"file_name": "contract_A.pdf"},
    n_results=10
)

ChromaDB file structure

data/chroma_db/
├── chroma.sqlite3              # Main database (metadata, IDs)
├── 00000000-0000-0000-0000-000000000000/  # Collection directory
│   ├── data_level0.bin         # HNSW layer 0
│   ├── data_level1.bin         # HNSW layer 1
│   ├── length.bin              # Vector lengths
│   └── link_lists.bin          # HNSW links
└── ...

3. BM25 index schema (sparse retrieval)

File format

File: data/bm25_index/bm25_index.pkl (Python pickle)

Structure

python{
  "bm25": BM25Okapi,              # BM25 model instance
  "chunk_ids": List[str],         # Chunk IDs (parallel to docs)
  "documents": List[str],         # Chunk texts (tokenized)
  "metadata": {
    "version": "1.0",
    "created_at": "2026-08-03T14:23:45",
    "num_docs": 18820,
    "vocab_size": 47234
  }
}

BM25Okapi parameters

pythonfrom rank_bm25 import BM25Okapi

bm25 = BM25Okapi(
    corpus=tokenized_docs,  # List[List[str]]
    k1=1.5,                 # Term frequency saturation
    b=0.75,                 # Length normalization
    epsilon=0.25            # IDF floor
)

Tokenization

Simple whitespace tokenization:

pythondef tokenize(text: str) -> List[str]:
    """BM25 tokenizer."""
    return text.lower().split()

Querying BM25

pythonimport pickle

with open("data/bm25_index/bm25_index.pkl", "rb") as f:
    data = pickle.load(f)

bm25 = data["bm25"]
chunk_ids = data["chunk_ids"]

# Query
query_tokens = tokenize("What is rescission?")
scores = bm25.get_scores(query_tokens)

# Top-K
top_k_indices = np.argsort(scores)[::-1][:50]
top_k_chunk_ids = [chunk_ids[i] for i in top_k_indices]
top_k_scores = [scores[i] for i in top_k_indices]

4. Workspace history schema (per-project)

File format

File: <workspace>/workspace/history.json

Encoding: UTF-8

Format: JSON (pretty-printed, 2-space indent)

Schema

json{
  "conversations": [
    {
      "id": "1722567890123",
      "title": "Contract Law Questions",
      "messages": [
        {
          "id": "msg_1",
          "role": "user",
          "content": "What is rescission?",
          "timestamp": "2026-08-03T14:23:45.123Z"
        },
        {
          "id": "msg_2",
          "role": "assistant",
          "content": "Rescission is the unmaking of a contract...",
          "timestamp": "2026-08-03T14:23:52.456Z",
          "sources": [
            {
              "file_name": "contract_law.pdf",
              "page": 12,
              "section": "Remedies",
              "score": 0.847,
              "text": "Rescission is the unmaking..."
            }
          ],
          "metrics": {
            "totalMs": 7123,
            "approvalLatencyMs": 0,
            "retryCount": 0,
            "nodeTimings": {}
          },
          "answer_found": true
        }
      ],
      "created_at": "2026-08-03T14:23:45.123Z",
      "updated_at": "2026-08-03T14:23:52.456Z"
    }
  ],
  "activeConversationId": "1722567890123",
  "working_set": [
    {
      "file_name": "contract_law.pdf",
      "page": 12,
      "section": "Remedies",
      "text": "Rescission is...",
      "pinned": true,
      "timestamp": "2026-08-03T14:24:00.000Z"
    }
  ],
  "gpu_profile": {
    "residency_mode": "adaptive",
    "keep_resident_free_mb": 1024,
    "hot_min_uses": 2,
    "hot_window_seconds": 120,
    "max_resident": 2
  }
}

Field descriptions

Conversations:

• id: Unix timestamp (milliseconds)
• title: Auto-generated from first user message (max 50 chars)
• messages: Chronological message list
• created_at, updated_at: ISO 8601 timestamps

Messages:

• role: "user" | "assistant"
• sources: Retrieved documents (only for assistant messages)
• metrics: Pipeline performance metrics
• answer_found: Boolean (abstain signal)

Working set:

• Pinned/selected sources from conversations
• pinned: User explicitly pinned
• timestamp: When added to working set

GPU profile:

• Per-project GPU residency settings
• See GPU_MANAGEMENT.md §7

5. LangGraph conversations schema (SQLite)

Database file

File: data/conversations.db

Engine: SQLite3

Schema

sqlCREATE TABLE checkpoints (
    thread_id TEXT NOT NULL,
    checkpoint_id TEXT NOT NULL,
    parent_id TEXT,
    checkpoint BLOB NOT NULL,
    metadata TEXT,
    created_at REAL NOT NULL,
    PRIMARY KEY (thread_id, checkpoint_id)
);

CREATE TABLE checkpoint_blobs (
    thread_id TEXT NOT NULL,
    checkpoint_id TEXT NOT NULL,
    channel TEXT NOT NULL,
    type TEXT NOT NULL,
    blob BLOB NOT NULL,
    PRIMARY KEY (thread_id, checkpoint_id, channel)
);

CREATE TABLE checkpoint_writes (
    thread_id TEXT NOT NULL,
    checkpoint_id TEXT NOT NULL,
    task_id TEXT NOT NULL,
    idx INTEGER NOT NULL,
    channel TEXT NOT NULL,
    type TEXT,
    blob BLOB,
    PRIMARY KEY (thread_id, checkpoint_id, task_id, idx)
);

Checkpoints table

Thread lifecycle

1. New conversation → thread_id created
2. Each tool call → new checkpoint saved
3. Human-in-the-loop interrupt → checkpoint persists, thread pauses
4. Resume → load checkpoint, continue graph execution
5. Conversation ends → checkpoints remain (for future resume)

Querying conversations

pythonimport sqlite3

conn = sqlite3.connect("data/conversations.db")
cursor = conn.cursor()

# List all threads
cursor.execute("SELECT DISTINCT thread_id FROM checkpoints")
threads = cursor.fetchall()

# Get latest checkpoint for a thread
cursor.execute("""
    SELECT checkpoint_id, created_at
    FROM checkpoints
    WHERE thread_id = ?
    ORDER BY created_at DESC
    LIMIT 1
""", (thread_id,))

6. Compliance flags schema (SQLite)

Database file

File: data/flags.db

Engine: SQLite3

Schema

sqlCREATE TABLE flags (
    flag_id TEXT PRIMARY KEY,
    document_id TEXT NOT NULL,
    chunk_id TEXT,
    section_title TEXT,
    page_number INTEGER,
    rule_id TEXT NOT NULL,
    rule_description TEXT NOT NULL,
    severity TEXT NOT NULL,
    clause_text TEXT NOT NULL,
    reason TEXT,
    confidence REAL,
    status TEXT NOT NULL DEFAULT 'open',
    created_at REAL NOT NULL,
    updated_at REAL NOT NULL,
    resolved_at REAL,
    resolved_by TEXT
);

CREATE INDEX idx_document_id ON flags(document_id);
CREATE INDEX idx_status ON flags(status);
CREATE INDEX idx_severity ON flags(severity);

Flags table

Flag lifecycle

1. Scan document → flags created with status="open"
2. User reviews → status changes:
   • "resolved" — issue fixed
   • "dismissed" — false positive
   • "escalated" — needs legal review
3. Resolved flags → resolved_at timestamp set

Querying flags

pythonimport sqlite3

conn = sqlite3.connect("data/flags.db")
cursor = conn.cursor()

# All open high-severity flags
cursor.execute("""
    SELECT flag_id, document_id, rule_description, clause_text
    FROM flags
    WHERE status = 'open' AND severity = 'high'
    ORDER BY created_at DESC
""")

# Flags for a specific document
cursor.execute("""
    SELECT flag_id, severity, rule_description
    FROM flags
    WHERE document_id = ?
    ORDER BY severity, page_number
""", (document_id,))

7. Frontend localStorage schema

Storage keys (per-project scoped)

Frontend uses per-project scoped localStorage keys:

Hash suffix: SHA256 of workspace path → per-project isolation.

Chat storage schema

Key: freebuff-chat-storage:<hash>

json{
  "state": {
    "conversations": [ /* same as history.json */ ],
    "activeConversationId": "1722567890123",
    "sources": [ /* current query sources */ ],
    "streamingAnswer": "",
    "isStreaming": false,
    "approvalPending": false
  },
  "version": 1
}

Zustand persist middleware auto-syncs to localStorage on state changes.

Working set schema

Key: freebuff-working-set:<hash>

json{
  "sources": [
    {
      "file_name": "contract.pdf",
      "page": 3,
      "section": "Payment Terms",
      "text": "Payment is due...",
      "pinned": true,
      "starred": false
    }
  ]
}

Pipeline layout schema

Key: pipeline-layout-v1 (global, not per-project)

json{
  "version": 1,
  "nodes": [
    {
      "id": "dense",
      "type": "denseRetrieval",
      "position": {"x": 280, "y": 440},
      "data": {"label": "Dense Retrieval"}
    }
  ],
  "edges": [
    {
      "id": "planner-dense",
      "type": "particle",
      "source": "planner",
      "target": "dense"
    }
  ]
}

See PIPELINE_JSON.md for full schema.

8. Log file format

File location

File: logs/legal_retrieval.log

Rotation: 10 MB per file, 5 backups

Format

YYYY-MM-DD HH:MM:SS | LEVEL   | module.name | message

Example:

2026-08-03 14:23:45 | INFO    | legal_retrieval.retrieval | Query: What is rescission?
2026-08-03 14:23:46 | DEBUG   | legal_retrieval.embedder | Embedding batch size: 32
2026-08-03 14:23:50 | INFO    | legal_retrieval.retrieval | Retrieved 50 results in 4.3s
2026-08-03 14:23:52 | WARNING | legal_retrieval.reranker | Reranker score below threshold: 0.18
2026-08-03 14:23:53 | ERROR   | legal_retrieval.llm | Ollama connection failed: [Errno 111]

Log levels

Parsing logs

bash# All errors
grep "ERROR" logs/legal_retrieval.log

# Query latency
grep "Retrieved.*results" logs/legal_retrieval.log | \
  grep -oP "\d+\.\d+s"

# Today's logs
awk -v today="$(date +%Y-%m-%d)" '$1 == today' logs/legal_retrieval.log

9. Model storage

BGE-M3 (embedder)

Location: models/bge-m3/

models/bge-m3/
├── config.json           # Model config (hidden size, num layers, etc.)
├── pytorch_model.bin     # Model weights (~1.1 GB)
├── tokenizer_config.json # Tokenizer settings
├── vocab.txt             # Vocabulary (32k tokens)
└── ...

BGE-Reranker-v2-M3

Location: models/bge-reranker-v2-m3/

models/bge-reranker-v2-m3/
├── config.json           # Model config
├── pytorch_model.bin     # Model weights (~2.2 GB)
├── tokenizer_config.json
├── vocab.txt
└── ...

Model download

First run downloads models from HuggingFace:

pythonfrom sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "BAAI/bge-m3",
    cache_folder="models/bge-m3",
)

10. Temporary file storage

Processed directory

Location: <workspace>/workspace/processed/ or data/processed/

Contents:

• Docling temporary images (PDF page images for OCR)
• Intermediate parsing artifacts
• Backup embeddings (crash recovery)

Cleanup:

• Auto-cleaned on restart (old files removed)
• Not included in workspace backups

11. Data migrations

ChromaDB version migration

When upgrading ChromaDB:

pythonfrom chromadb import PersistentClient

# Old version
old_client = PersistentClient(path="data/chroma_db_old")
old_collection = old_client.get_collection("legal_chunks")

# New version
new_client = PersistentClient(path="data/chroma_db")
new_collection = new_client.create_collection("legal_chunks")

# Migrate data
all_data = old_collection.get()
new_collection.add(
    ids=all_data["ids"],
    embeddings=all_data["embeddings"],
    documents=all_data["documents"],
    metadatas=all_data["metadatas"]
)

History.json schema migration

Add new fields while preserving old data:

pythonimport json

with open("workspace/history.json") as f:
    data = json.load(f)

# Migrate: add gpu_profile if missing
if "gpu_profile" not in data:
    data["gpu_profile"] = {
        "residency_mode": "adaptive",
        "keep_resident_free_mb": 1024,
        "hot_min_uses": 2,
        "hot_window_seconds": 120,
        "max_resident": 2
    }

with open("workspace/history.json", "w") as f:
    json.dump(data, f, indent=2)

12. Backup strategies

Workspace backup

Backup entire workspace (per-project):

bashtar -czf project-backup-$(date +%Y%m%d).tar.gz \
  /path/to/project/workspace/

Includes:

• ChromaDB index
• BM25 index
• Chat history + GPU profile
• Processed temp files

Global data backup

Backup global databases:

bashtar -czf global-backup-$(date +%Y%m%d).tar.gz \
  data/conversations.db \
  data/flags.db \
  logs/

Model backup

Models are re-downloadable — backup not critical:

bashtar -czf models-backup.tar.gz models/

13. Storage quotas & limits

14. Related documentation