API Provider Mode — run Lawyer Assistant on remote AI

Local mode is 100% unaffected: unless a config file explicitly enables API mode, the app behaves exactly as before (Ollama LLM + BGE-M3 + BGE-Reranker).

1. How it manages your models

The ChromaDB and BM25 indexes still live inside your project folder's workspace/ directory, so switching projects works exactly the same.

2. Enabling API mode

Copy backend/api_config.sample.json to backend/api_config.json, set "enabled": true, fill in your key/model, and restart the backend:

json{
  "enabled": true,
  "provider": "openai",
  "base_url": "https://api.openai.com/v1",
  "api_key": "sk-...",
  "llm_model": "gpt-4o-mini",
  "embed_model": "text-embedding-3-small",
  "embed_dim": 0,
  "rerank_mode": "auto",
  "temperature": 0.3,
  "max_tokens": 4096,
  "timeout": 120,
  "thinking_budget": 0
}

backend/api_config.json is git-ignored — the API key never gets committed. You can also point at a different file with the LAWYER_API_CONFIG env var.

thinking_budget enables Anthropic extended thinking (see the Anthropic section below). 0 = off, the default.

Environment-variable overrides

Every field can be overridden by an env var (these win over the JSON file):

3. Providers

openai / openai_compatible — any OpenAI-compatible API: OpenAI, Groq, DeepSeek, Mistral, OpenRouter, Together, Jina, Moonshot, Fireworks, xAI, ... Chat, embeddings and function calling all work.

gemini — Google Gemini (chat + embeddings). Set base_url to your regional endpoint if you use one (e.g. https://generativelanguage.googleapis.com/v1beta is the default).

cohere — Cohere (chat + embeddings + rerank).

anthropic — Anthropic Claude via the Messages API (chat + streaming with extended thinking). Thinking blocks stream through the same Chain-of-Thought channel as OpenAI-compatible reasoning models, so the CoT panel works identically. Set api_key to your Anthropic key (sk-ant-...); the default base_url is https://api.anthropic.com/v1.

Extended thinking (optional): set "thinking_budget": 2048 (or LAWYER_THINKING_BUDGET=2048) to enable Claude's extended thinking. The budget must stay below max_tokens (it is clamped automatically, and floored at 1024 tokens per Anthropic's requirement — when max_tokens leaves enough headroom). While thinking is on, temperature is omitted from requests — the Anthropic API rejects sending them together.

Anthropic has no embeddings API — to use Claude for the LLM with remote embeddings, configure an embed_base_url/embed_model override (Jina, Voyage, OpenAI, ...), or keep embeddings local. Agent-mode tool calling is not supported yet for Anthropic (same as Gemini/Cohere) — use an OpenAI-compatible provider for the LangGraph agent.

Mixing providers

llm_base_url/llm_api_key and embed_base_url/embed_api_key let you mix — e.g. a DeepSeek LLM with Jina embeddings:

json{
  "enabled": true,
  "provider": "openai_compatible",
  "base_url": "https://api.deepseek.com/v1",
  "api_key": "sk-deepseek...",
  "llm_model": "deepseek-chat",
  "embed_base_url": "https://api.jina.ai/v1",
  "embed_api_key": "jina_...",
  "embed_model": "jina-embeddings-v3",
  "rerank_mode": "jina"
}

Note: DeepSeek/Groq/OpenRouter do not offer embeddings — use an embed_base_url override (Jina, Voyage, OpenAI, ...) or stay local for embeddings.

4. Reranking modes

• auto — use Cohere rerank when a rerank endpoint is configured, otherwise fall back to embedding-similarity reranking (query embedding vs candidate embeddings, no extra API beyond embeddings).
• cohere — Cohere /rerank endpoint.
• jina — Jina /rerank endpoint (OpenAI-style).
• embedding — cosine similarity rerank using the configured embedder.
• none — keep retrieval order (no rerank).

5. Streaming & chain-of-thought

The full SSE chat flow (planner → tool call → answer) works over APIs. Thinking-capable OpenAI-compatible models that stream reasoning_content (DeepSeek-R1, Qwen3, ...) feed the same Chain-of-Thought panel as local thinking models. Anthropic extended-thinking thinking_delta blocks are mapped to the same reasoning_content channel, so Claude's deliberation shows up identically in the CoT panel.

The legacy LangGraph agent with human-in-the-loop approval works too, but tool calling is only supported on OpenAI-compatible providers (Gemini/Cohere/ Anthropic raise a clear error for agent mode).

6. Checking the mode is active

bashcurl http://127.0.0.1:8765/api/providers/status

Returns which provider/models are active. The API key is never returned. The GPU status endpoint will show nothing loaded in API mode — that is expected (no local models are resident).

7. Troubleshooting

• "API request failed (401/403)" — wrong/missing key; check the config file path is being read (config_source in the status endpoint).
• "embedding dim mismatch" — your embed model's dimension differs from embed_dim. Set embed_dim (OpenAI text-embedding-3-small = 1536, text-embedding-3-large = 3072, Gemini text-embedding-004 = 768) or leave it 0 to auto-detect.
• Anthropic: "no embedding model is configured" — Anthropic has no embeddings API. Set embed_base_url/embed_model (Jina, Voyage, OpenAI, ...) or keep embeddings local (remove embed_model/embed_base_url and rely on BGE-M3).
• Switching modes on an existing project — the ChromaDB collection keeps the dimension it was built with. Re-ingest the project after changing embedding providers.
• Slow first query — the embedder probes the dimension once at startup; afterwards everything is cached.