Workspace Management & Auto-Ingestion

1. What is a workspace?

A workspace is a user-picked project folder that becomes the active context for all operations:

• Uploads land directly in the folder (raw files stay visible as project documents)
• Indexes live under workspace/ (chroma_db/, bm25_index/, processed/)
• Chat history is per-project (workspace/history.json)
• GPU settings are per-project (residency profile in history.json)

Before workspace system (legacy)

Everything lived in a static data/ directory — one index for all documents, mixed chat history, no project separation.

After workspace system (current)

Each folder is self-contained. Point the app at a different folder and you get that folder's index, history, and settings — no cross-contamination.

2. Selecting a workspace

In the desktop app

Click "Work in a folder" in the sidebar → a native Electron directory dialog opens → pick your project folder. The backend receives the absolute path (no file re-upload needed).

Via API

bash# Set workspace
curl -X POST http://localhost:8765/api/workspace \
  -H "Content-Type: application/json" \
  -d '{"path": "/absolute/path/to/project"}'

# Get current workspace
curl http://localhost:8765/api/workspace

# Clear workspace (reset to no active workspace)
curl -X DELETE http://localhost:8765/api/workspace

Requirements

A workspace is mandatory for:

• Chat (/api/chat, /api/chat/stream, /api/chat/resume)
• File uploads (/api/upload)
• Compliance scans (/api/scan/*)

These endpoints return "No workspace selected. Select a project folder first." when none is active — there is no default data/ folder fallback.

3. Workspace storage layout

When a workspace is active, storage resolves as follows:

/path/to/your-project/
├── contract_A.pdf            ← raw uploads land here (visible as project docs)
├── agreement_B.docx          ← visible to the user
├── memo.txt
└── workspace/                ← private data folder (created automatically)
    ├── chroma_db/            ← vector index (per project)
    ├── bm25_index/           ← sparse index
    ├── processed/            ← temp artifacts (Docling images, etc.)
    └── history.json          ← chat history + working set + GPU profile

No workspace active → falls back to project data/ directory:

project-root/
└── data/
    ├── raw/
    ├── chroma_db/
    ├── bm25_index/
    └── processed/

Workspace override is runtime-only (resets on server restart or DELETE /api/workspace).

4. Background folder watcher (auto-ingestion)

The watcher is a daemon thread that polls the active workspace folder for new documents and auto-ingests them — drop a contract into the folder and it becomes searchable within seconds, with zero user action.

How it works

┌────────────────────────────────────────────────────────────┐
│                   FOLDER WATCHER LOOP                       │
└────────────────────────────────────────────────────────────┘

1. Every 8 seconds (configurable): compute folder fingerprint
   ├─ {relative_path: (size, mtime_ns)} for supported documents
   ├─ Excludes workspace/ storage dirs
   ├─ Excludes temp/lock files (~$*, .~lock*, *.tmp, *.bak, ...)
   └─ Cheap (no ChromaDB call)

2. Compare fingerprint to last scan:
   ├─ No change → skip cycle (idle, no disk/GPU work)
   └─ Changed → proceed to step 3

3. Acquire INGEST_LOCK (non-blocking):
   ├─ Manual SSE ingest in flight? → skip cycle, retry next poll
   └─ Lock acquired → proceed to step 4

4. Run stream_ingest_directory():
   ├─ Docling parses new files
   ├─ StructureAwareChunker splits them
   ├─ BGE-M3 embeds chunks
   ├─ ChromaDB stores vectors
   └─ BM25 index rebuilt

5. Commit fingerprint → next cycle (8s later)

Key features

Polling, not fs events

• No new dependency (watchdog)
• Works on all platforms (Windows, macOS, Linux, network drives)
• Trivially safe to restart
• Configurable interval (default 8 seconds)

Cheap skip detection

• Fingerprint comparison is file metadata only (size + mtime)
• When nothing changed, zero ChromaDB calls
• Only changed fingerprints trigger ingestion

Serialization with manual ingestion

• INGEST_LOCK shared by watcher + SSE manual ingest
• ChromaDB writes never race
• Watcher skips a cycle (non-blocking acquire) when manual ingest is running

Self-stopping on workspace switch

• Thread checks get_workspace_root() every cycle
• Auto-stops if the active workspace changed (handles missed teardowns)

Status exposed to UI

• GET /api/workspace/watch/status returns live watcher state

5. Watcher status API

bashcurl http://localhost:8765/api/workspace/watch/status

Returns:

json{
  "watch": {
    "watching": true,
    "root": "/absolute/path/to/project",
    "last_scan_at": 1750000000.0,
    "last_scan_duration_ms": 45,
    "files_found": ["contract.pdf", "agreement.docx"],
    "files_indexed": ["contract.pdf"],
    "chunks_indexed": 23,
    "skipped": 1,
    "last_error": null
  }
}

Fields:

• watching — whether the watcher thread is running
• root — the folder being watched
• last_scan_at — Unix timestamp of last scan
• last_scan_duration_ms — how long the scan took
• files_found — all supported docs discovered
• files_indexed — files actually indexed (excludes already-present ones)
• chunks_indexed — total chunks added
• skipped — already-indexed file count
• last_error — last exception message (null when healthy)

6. Manual workspace ingestion

Trigger a one-time ingestion of every supported document in the workspace:

Non-streaming (blocking)

bashcurl -X POST http://localhost:8765/api/workspace/ingest

Returns when done:

json{
  "chunks": 458,
  "path": "/absolute/path/to/project",
  "elapsed_ms": 12340
}

Streaming (SSE with progress)

bashcurl -X POST http://localhost:8765/api/workspace/ingest/stream

Streams events:

event: ingest_start
data: {"type":"ingest_start","files":["a.pdf","b.docx"],"total":2,"skipped":0}

event: ingest_progress
data: {"type":"ingest_progress","file":"a.pdf","current":1,"total":2,"chunks":23}

event: ingest_progress
data: {"type":"ingest_progress","file":"b.docx","current":2,"total":2,"chunks":45}

event: ingest_done
data: {"type":"ingest_done","chunks":68,"skipped":0,"elapsed_ms":3456}

Force re-index

Add ?force=1 to re-index already-indexed files (rebuilds the entire index):

bashcurl -X POST "http://localhost:8765/api/workspace/ingest/stream?force=1"

Cancel streaming ingestion

bashcurl -X POST http://localhost:8765/api/workspace/ingest/cancel

Stops cleanly at the next file boundary (already-completed files stay indexed).

7. Workspace file listing

Get discovered documents in the active workspace (UI "workspace panel"):

bashcurl http://localhost:8765/api/workspace

Returns:

json{
  "active": true,
  "path": "/absolute/path/to/project",
  "name": "project",
  "workspace_dir": "/absolute/path/to/project/workspace",
  "gpu_profile": { "residency_mode": "adaptive", ... },
  "files": [
    { "name": "contract_A.pdf", "size": 123456, "modified": 1750000000.0 },
    { "name": "agreement_B.docx", "size": 67890, "modified": 1750000100.0 }
  ]
}

Files are:

• Capped at 50 documents (UI limit)
• Filtered to supported extensions (see SUPPORTED_EXTENSIONS)
• Excludes workspace/ storage dirs
• Excludes temp/lock files

8. Configuration

All watcher configuration is in backend/legal_retrieval/workspace_watcher.py:

Note: The poll interval is hardcoded, not an env var. To change it, pass poll_interval=<seconds> when constructing WorkspaceWatcher() (or modify DEFAULT_POLL_INTERVAL directly).

9. Per-project GPU profiles

Each workspace carries its own GPU residency profile (mode, hot window, max resident, ...) persisted in workspace/history.json under "gpu_profile".

bash# Get active project's GPU profile
curl http://localhost:8765/api/gpu/profile

# Apply and persist a profile to the active project
curl -X POST http://localhost:8765/api/gpu/profile \
  -H "Content-Type: application/json" \
  -d '{"residency_mode":"persistent","max_resident":3}'

When you switch projects (POST /api/workspace), the new project's GPU profile is automatically loaded — settings never leak between projects.

See GPU_MANAGEMENT.md §7 for GPU profile details.

10. Workspace history (chat + working set + GPU)

workspace/history.json stores:

• conversations — per-project chat history
• activeConversationId — last-selected conversation
• starred_messages — user-starred messages (optional)
• working_set — pinned sources
• gpu_profile — per-project GPU residency settings

The frontend auto-saves this file via POST /api/workspace/history:

bashcurl -X POST http://localhost:8765/api/workspace/history \
  -H "Content-Type: application/json" \
  -d @history.json

Read it back:

bashcurl http://localhost:8765/api/workspace/history

Returns:

json{
  "history": {
    "conversations": [ ... ],
    "gpu_profile": { ... }
  }
}

Schema:

• conversations — array of conversation objects (messages, sources, ...)
• gpu_profile — GPU residency settings (see GPU_PROFILE_FIELDS in config.py)

Frontend keys (freebuff-chat-storage, freebuff-working-set) are scoped per project via a hash suffix — switching workspaces in the UI re-points the localStorage keys, and the server-side history.json acts as the durable backup.

11. Implementation details

Watcher lifecycle

• Started — POST /api/workspace starts the watcher on the selected folder
• Stopped — DELETE /api/workspace stops it
• Self-stops — if get_workspace_root() changes out from under it (leaked thread guard)

Thread safety

• _lock protects _status (the snapshot returned by status())
• INGEST_LOCK serializes ChromaDB writes (shared with manual ingest)
• Non-blocking acquire — watcher skips a cycle when manual ingest is running

12. Troubleshooting

13. Why polling, not fs events?

For a desktop app with human-paced workflows (dropping contracts), 8-second latency is imperceptible — the simplicity/reliability trade-off favors polling.

14. Future enhancements

Planned but not yet implemented:

• Recursive subdirectory support — currently only top-level files are watched
• Configurable poll interval via env var (PLR_WATCHER_POLL_INTERVAL)
• Watcher pause/resume — UI button to temporarily disable auto-ingest
• Per-file progress in watcher status — currently only totals
• Selective file-type watching — watch only PDFs, ignore images, etc.

See PIPELINE_ROADMAP.md for other planned features.

15. Related documentation