Pipeline Layout JSON — spec for AI models & hand-editors

• Ground truth code: frontend/src/components/Pipeline/pipelineStorage.ts (serialize / parse / validate), defaults.ts (default nodes & edges), nodes.tsx (node types + bypass edges).
• Feature overview: PIPELINE_EDITOR.md.
• Backend wiring: since the /api/pipeline/config endpoint was added, this JSON now also reconfigures retrieval at runtime (reranking, search mode, top_k, dense/sparse weights). See §10.

1. File envelope (two accepted shapes)

The import function parsePipelineLayout(text) accepts either:

jsonc// Shape A — versioned export (what the editor's ↓ button writes)
{
  "version": 1,
  "nodes": [ /* Node[] */ ],
  "edges": [ /* Edge[] */ ]
}

jsonc// Shape B — bare layout (also valid; version is optional)
{
  "nodes": [ /* Node[] */ ],
  "edges": [ /* Edge[] */ ]
}

If you author a file, prefer Shape A with "version": 1 (matching serializeLayout). version is informational — the parser does not check it.

2. Node object schema

jsonc{
  "id": "dense",                        // string, REQUIRED, must be unique across nodes
  "type": "denseRetrieval",             // string, REQUIRED, must be in KNOWN_TYPES (see §3)
  "position": { "x": 280, "y": 440 },   // {x,y} numbers, REQUIRED (any values allowed)
  "data": {
    "label": "Dense Retrieval",         // string, recommended (shown on the node)
    "status": "idle",                   // optional, one of idle|active|done|error; runtime-only
    "files": ["a.pdf"],                 // optional, ingest node only
    "icon": "🔧",                       // optional, tool node only
    "description": "..."                // optional, tool node only
  }
}

Rules enforced by validateLayout:

• type must be one of the 10 known types (§3). Unknown/stale types are dropped silently (e.g. the pre-split searchDocuments / scanIngest).
• Duplicate node ids: only the first occurrence is kept.
• Runtime-only fields (status, active, selected) are tolerated on import and overridden at render (the editor re-derives them); they are stripped on export. You may include them or omit them.

3. Valid node types (KNOWN_TYPES)

The five toggleable sub-nodes (palette switches) are: denseRetrieval, sparseRetrieval, rerank, ingest, scan — mapped to canonical ids dense, sparse, rerank, ingest, scan. Keep those ids if you want the toggle switches to control the node.

4. Edge object schema

jsonc{
  "id": "dense-rerank",                 // string, REQUIRED, must be unique across edges
  "type": "particle",                   // string; 'particle' or 'default' kept, anything else coerced to 'particle'
  "source": "dense",                    // string, REQUIRED — must exist in nodes, else edge dropped
  "target": "rerank",                   // string, REQUIRED — must exist in nodes, else edge dropped
  "sourceHandle": "chitchat",           // optional — named output handle (see §5)
  "targetHandle": null,                 // optional — usually omitted
  "label": "candidates",                // optional — shown on the edge
  "style": { "stroke": "#8b5cf6", "strokeWidth": 2 },           // optional
  "labelStyle": { "fill": "#8b5cf6", "fontSize": 11, "fontWeight": 600 },  // optional
  "labelBgStyle": { "fill": "var(--bg-primary)", "fillOpacity": 0.9 },     // optional
  "data": {
    "color": "#8b5cf6",                 // optional — used by the particle animation
    "active": false,                    // optional — runtime-only, stripped on export
    "dataType": "candidates",           // optional — shown in live payload preview
    "payloadPreview": "top 50 dense"    // optional — shown in live payload preview
  }
}

Rules enforced by validateLayout:

• Edge is dropped if source or target does not reference an existing node id (that survived validation).
• Edge is dropped if id is missing or a duplicate (first occurrence kept).
• type other than 'particle'/'default' is coerced to 'particle'.

5. Named handles (important for the intent node)

Only the intent node exposes named source handles. If an edge leaves intent, set sourceHandle to one of:

All other nodes use a single unnamed source/target handle — omit sourceHandle/targetHandle for them.

6. Bypass edges (what a removed sub-node looks like)

When a middle sub-node is removed, the editor rewires incoming edges straight to outgoing targets. Those rewires are tagged so re-enabling can find them:

• id pattern: bypass-<removedNodeId>-<incSource>-<outTarget>-<i>-<j>-<timestamp> (e.g. bypass-rerank-dense-answer-0-0-1750000000000)
• type: "particle", label: "bypass", data: { color: "var(--accent)", active: false, dataType: "data", payloadPreview: "bypassed step" }
• source/target are the surviving nodes; sourceHandle/targetHandle are carried over from the incoming edge.

Example — the reranker removed:

jsonc{ "id": "bypass-rerank-dense-answer-0-0-1750000000000", "type": "particle",
  "source": "dense", "target": "answer", "label": "bypass",
  "data": { "color": "var(--accent)", "active": false, "dataType": "data",
  "payloadPreview": "bypassed step" } }
{ "id": "bypass-rerank-sparse-answer-1-0-1750000000000", "type": "particle",
  "source": "sparse", "target": "answer", "label": "bypass",
  "data": { "color": "var(--accent)", "active": false, "dataType": "data",
  "payloadPreview": "bypassed step" } }

7. A complete valid example (dense + sparse → answer, no rerank, no scan)

This file is guaranteed to import cleanly — it references only known node types, unique ids, and edges whose endpoints exist:

json{
  "version": 1,
  "nodes": [
    { "id": "intent", "type": "intentClassifier", "position": { "x": 450, "y": 0 }, "data": { "label": "Intent Classifier" } },
    { "id": "planner", "type": "planner", "position": { "x": 450, "y": 220 }, "data": { "label": "Planner" } },
    { "id": "dense", "type": "denseRetrieval", "position": { "x": 280, "y": 440 }, "data": { "label": "Dense Retrieval" } },
    { "id": "sparse", "type": "sparseRetrieval", "position": { "x": 620, "y": 440 }, "data": { "label": "Sparse Retrieval" } },
    { "id": "answer", "type": "answer", "position": { "x": 450, "y": 880 }, "data": { "label": "Answer" } }
  ],
  "edges": [
    { "id": "intent-planner", "type": "particle", "source": "intent", "target": "planner", "sourceHandle": "planner",
      "label": "legal query", "style": { "stroke": "#f59e0b", "strokeWidth": 2 },
      "data": { "color": "#f59e0b", "active": false, "dataType": "user query", "payloadPreview": "What is rescission?" } },
    { "id": "planner-dense", "type": "particle", "source": "planner", "target": "dense", "label": "query",
      "style": { "stroke": "#8b5cf6", "strokeWidth": 2 },
      "data": { "color": "#8b5cf6", "active": false, "dataType": "query", "payloadPreview": "dense search on embeddings" } },
    { "id": "planner-sparse", "type": "particle", "source": "planner", "target": "sparse", "label": "query",
      "style": { "stroke": "#8b5cf6", "strokeWidth": 2 },
      "data": { "color": "#8b5cf6", "active": false, "dataType": "query", "payloadPreview": "BM25 keyword search" } },
    { "id": "bypass-rerank-dense-answer-0-0-1750000000000", "type": "particle", "source": "dense", "target": "answer",
      "label": "bypass", "style": { "stroke": "var(--accent)", "strokeWidth": 2 },
      "data": { "color": "var(--accent)", "active": false, "dataType": "data", "payloadPreview": "bypassed step" } },
    { "id": "bypass-rerank-sparse-answer-1-0-1750000000000", "type": "particle", "source": "sparse", "target": "answer",
      "label": "bypass", "style": { "stroke": "var(--accent)", "strokeWidth": 2 },
      "data": { "color": "var(--accent)", "active": false, "dataType": "data", "payloadPreview": "bypassed step" } }
  ]
}

Note: bypass-rerank-… ids are arbitrary here — the editor only matches them by the bypass-<nodeId>- prefix when re-enabling the reranker. Using the real pattern keeps re-enable behavior correct.

8. Generation checklist (AI must follow all of these)

1. Envelope: { "version": 1, "nodes": [...], "edges": [...] }.
2. Every node: has string id (unique), valid type from §3, and a position object with numeric x/y. Recommended: data.label.
3. Every edge: has string id (unique), source and target that both exist in the node list, and type: "particle".
4. No dangling edges: an edge to a node you deleted is invalid — it will be silently dropped.
5. No unknown types: using searchDocuments, scanIngest, or any other non-§3 type gets the node silently dropped. This is the most common AI mistake.
6. Named handles only from intent (§5); omit them elsewhere.
7. Zero nodes is legal (a deliberately emptied canvas round-trips), but a file with edges and no valid nodes is pointless — edges get dropped.
8. Keep ids stable and semantic (dense, sparse, rerank, ingest, scan, planner, answer, intent, chitchat) so palette toggles and re-enabling work.
9. Extra fields are tolerated: the parser ignores unknown properties.

9. Round-trip guarantee

Export writes {version, nodes, edges} with runtime fields stripped (status, active, selected). Import accepts that exact output back — export → import is identity-preserving. If you want to edit an exported file: change positions, delete nodes and every edge touching them, and keep all other fields as-is.

10. Backend runtime configuration (POST /api/pipeline/config)

The backend validates and consumes exactly the same JSON this spec covers. The editor no longer needs to be visualization-only: sending the layout to the backend reconfigures retrieval for every subsequent query.

Endpoint

• POST /api/pipeline/config — body is the layout JSON (Shape A or B from §1). Applies it at runtime.
• GET /api/pipeline/config — returns the currently active config.
• DELETE /api/pipeline/config — restore file defaults.

Optional query overrides on POST (win over the node-derived values): top_k (1–100), top_k_retrieval (1–500), dense_weight, sparse_weight (0–10).

bashcurl -X POST http://localhost:8765/api/pipeline/config \
  -H "Content-Type: application/json" \
  -d @pipeline-layout.json \
  "?top_k=8&dense_weight=0.7&sparse_weight=0.3"

Derivation rules (from which sub-nodes are present)

Precedence: file defaults < layout config < explicit per-request params (skip_rerank, top_k in the chat request). One exception: a layout with an explicit top_k override wins over the chat UI's fixed default of 5 (the frontend sends top_k: 5 unconditionally).

Behavior notes

• Validation reuses the same rules as the editor: unknown node types, dangling edges, and duplicate ids are dropped silently (§2–§4), so a file that imports cleanly into the editor also applies cleanly here.
• The response returns the derived config plus a validation summary (nodes_received/nodes_kept, edges_received/edges_kept, dropped_unknown_types, dropped_dangling_edges).
• skip_rerank, search_mode, top_k, and weights are read by RetrievalPipeline on construction (backend/legal_retrieval/pipeline_config.py), so every path — chat, tools, scan-upload ingestion queries — honors the layout automatically.
• Weights apply to the RRF fusion (rrf_score = dense_w/(k+rank_dense) + sparse_w/(k+rank_sparse)) and, when non-default, to the combine-and-dedup ordering for the no-rerank path.
• Config is runtime-only: it resets on server restart unless re-applied. Persistence lives in the editor's localStorage (pipeline-layout-v1) and the exported JSON file.
• ⚠️ Empty layout caveat: applying {nodes: [], edges: []} (a deliberately emptied canvas) derives skip_rerank=True — reranking is disabled globally until another layout or DELETE /api/pipeline/config restores it. Use DELETE (or the editor's Reset button, which is visual only) if you want retrieval untouched.