# Feature: Durable Background Execution + Human-in-the-Loop (HITL)

> Generated on: 2026-06-23
> Status: Draft
> Owner: Engineering Team
> Related PRs: #13633 (HITL v2); Epic LE-1437
> Companion document: [`../../CZL/HITL_FEATURE_OVERVIEW.md`](../../CZL/HITL_FEATURE_OVERVIEW.md) — the engineering deep-dive (8-stage code walk-through), and [`../../CZL/HITL_STATUS.md`](../../CZL/HITL_STATUS.md) — the status/decision summary.

---

## Table of Contents
1. [Overview](#1-overview)
2. [Ubiquitous Language Glossary](#2-ubiquitous-language-glossary)
3. [Domain Model](#3-domain-model)
4. [Behavior Specifications](#4-behavior-specifications)
5. [Architecture Decision Records](#5-architecture-decision-records)
6. [Technical Specification](#6-technical-specification)
7. [Observability](#7-observability)
8. [Deployment & Rollback](#8-deployment--rollback)
9. [Architecture Diagrams](#9-architecture-diagrams)

---

## 1. Overview

### Summary
Human-in-the-Loop (HITL) lets a flow **pause mid-run to ask a human to approve, reject, or supply input**, then resume exactly where it stopped. The pause is **durable**: it is checkpointed to the database, survives a process restart, and resumes without re-running or re-billing already-completed work. The paused/resumed run is fully **observable** as a single backend trace, with the approval step and the terminal output recorded as real spans.

### Business Context
Agentic flows take consequential actions (calling tools, sending requests, spending tokens). Teams need a control point where a human gates a risky step before it executes, without losing the run if the browser closes or the server restarts. HITL turns a flow into a long-lived, resumable workflow rather than a single fire-and-forget request.

### Bounded Context
Flow Execution — Durable Background Jobs & Graph Checkpointing.

### Related Contexts
- **Graph Engine** (Customer-Supplier): raises and restores the pause at layer boundaries.
- **Background Execution / Job Service** (Partnership): persists job status and serves the single-flight resume.
- **Tracing** (Conformist): records the run — including the gate and the terminal output — as spans.
- **Frontend (AG-UI)** (Customer-Supplier): reattaches to the live stream and renders the decision surfaces.

### Default-off guarantee
The pause probe is a no-op unless a component requests it, so normal flows are byte-for-byte unaffected: no checkpointing, no extra spans, no status churn.

---

## 2. Ubiquitous Language Glossary

| Term | Definition | Code Reference |
|------|------------|----------------|
| Pause request | A component's signal that it needs a human decision before continuing | `PauseRequested`, `GraphPausedException` |
| Checkpoint | A serialized snapshot of graph state at a layer boundary, written to the DB | `lfx/graph/checkpoint/schema.py`, `checkpoint` table |
| Suspend | The job transitioning to a durable waiting state | `JobStatus.SUSPENDED` |
| Resume | Re-hydrating the graph from its checkpoint and continuing past the pause | `resume_from_checkpoint`, `build_resumed_graph_and_get_order` |
| Single-flight resume | An atomic `SUSPENDED → IN_PROGRESS` claim so a decision is applied exactly once | `claim_suspended_for_resume` |
| Human input request | The pending question shown to the user (tool approval or HumanInput node) | `get_pending_human_request`, `human_input_required` event |
| Decision | The human's answer: `action_id` (e.g. `approve`/`reject`) plus optional `values` | `graph.human_input_decisions` |
| Run id | The graph's tracing identity; equals `graph_run_id` on the messages | `graph.set_run_id`, trace `id` |
| Job id | The durable job identity used for checkpoints + resume | `JobStatus`, `/api/v2/workflows/{job_id}/resume` |
| Gate span | The trace span recording the resolved decision: "Human In The Loop — {action label}" (e.g. Approve/Reject/Remove) | `TracingService.record_event_span` |

---

## 3. Domain Model

### 3.1 Aggregates

#### Durable Run Aggregate
- **Root Entity**: the background **Job** (`job_id`)
- **Entities**: the **Graph** (with its `run_id`), the **Checkpoint**, the **Pending Human Request**
- **Value Objects**: `JobStatus`, `Decision` (`action_id` + `values`), `request_id` (`{node_id}:{run_id}`; agent tool approvals append the LangGraph interrupt id — `{node_id}:{run_id}:{interrupt_id}` — so each approval in one run is individually addressable and a stale resume for approval N is rejected during approval N+1)
- **Invariants**:
  - A pause must persist a checkpoint before the run yields (no lost state).
  - Resume must be single-flight: a decision is applied **exactly once**.
  - Resume must not re-execute already-built vertices (no re-billing, no re-fired tools); dead branches stay dead.
  - The whole run (pre-pause + post-resume) must trace into **one** `run_id`.

### 3.2 Domain Events

| Event | Trigger | Payload | Consumers |
|-------|---------|---------|-----------|
| `human_input_required` | A component raises a pause | card/request descriptor | Frontend (renders card/bar); job runner (suspends) |
| Job `SUSPENDED` | Checkpoint written, run yields | `job_id`, `request_id` | Pending-requests API; UI badges |
| Resume accepted | Valid decision claimed | `job_id`, `decision` | Graph rebuild; tracing |
| Gate resolved span | Resume re-initializes tracing | "Human In The Loop — {decision}" | Trace panel (`/monitor/traces/{id}`) |
| Job `COMPLETED` / terminal span | Resumed run finishes | Chat Output span + flush | Trace panel; message history |

---

## 4. Behavior Specifications

### Feature: A flow pauses for a human decision and resumes durably

**As a** flow author
**I want** a step to pause for human approval and survive interruptions
**So that** risky actions are gated and no run is lost if the server restarts.

### Background
- Given a flow with an agent tool (or HumanInput node) that can request approval
- And durable background execution is enabled

### Scenario: Agent tool pauses for approval
- **Given** the agent decides to call a gated tool
- **When** the run reaches the tool call
- **Then** the graph checkpoints and the job becomes `SUSPENDED`
- **And** a `human_input_required` event surfaces an Approve/Reject card

### Scenario: Resume survives a process restart
- **Given** a job is `SUSPENDED` with a written checkpoint
- **When** the backend process restarts and the user approves
- **Then** the graph is rebuilt from the checkpoint and continues past the pause

### Scenario: Resume is not a re-run
- **Given** a paused run with already-built vertices
- **When** the run resumes
- **Then** built vertices are restored, not re-executed (no LLM re-billing, no re-fired tools)
- **And** a restored-built vertex is never handed back to the build loop by the runnable-predecessor walk (a re-run Chat Input would persist a duplicate `User` message for the turn)
- **And** branches killed before the pause stay dead

### Scenario: HITL inside a nested flow is rejected
- **Given** a flow whose Run Flow / Sub Flow / flow-as-tool target contains a connected Human Input node (or an approval-gated agent tool)
- **When** the nested flow is executed
- **Then** the run fails with a clear error ("cannot run as a nested flow… move the approval to the parent flow") instead of silently not pausing
- **And** a target flow with an unwired Human Input (no downstream consumer) still runs — it is skipped at runtime, not blocking

### Scenario: Two HITL nodes in sequence
- **Given** a flow with two HumanInput nodes where the first gates the path to the second
- **When** the first is approved, the second pauses, and the second is then approved
- **Then** the run finalizes once — no loop back to the first node
- **And** only the chosen branch of each node runs (the first node's non-chosen branches stay dead across the second pause's resume, not just within the build that answered them)

### Scenario: Rerunning a paused flow supersedes the stale pause
- **Given** a flow with a `SUSPENDED` run awaiting a decision
- **When** the same user submits a new background run of that flow under the same session
- **Then** the stale suspended run is cancelled (`CANCELLED`, pending request cleared) before the new run starts, so only the new pause is offered on every surface
- **And** the persisted chat card of the superseded pause is stamped `superseded` — after a history reload it renders closed ("Superseded by a new run") instead of turning interactive again and 409ing on every click
- **And** the scope is flow + user + effective session on the langflow backend (the submitting run's `session_id`, falling back to the flow id) — a rerun replaces stale pauses of the SAME session/thread, while SUSPENDED runs of other sessions stay untouched; the scope is flow on `lfx serve` (single API-key identity); running jobs are never superseded, so parallel runs stay supported

### Scenario: Supersede loses the race to a resume
- **Given** a `SUSPENDED` run whose resume arrives while a rerun's supersede is in flight
- **When** the resume wins the atomic `SUSPENDED → IN_PROGRESS` claim between supersede's select and its cancel
- **Then** supersede skips that job entirely — no `CANCELLED` write, no checkpoint delete, no metadata clear (backend) and no lingering STOP signal (`lfx serve`) — and the resumed run finishes normally

### Scenario: Single-flight resume
- **Given** two resume requests for the same `SUSPENDED` job
- **When** both arrive
- **Then** exactly one claims the `SUSPENDED → IN_PROGRESS` transition; the other is rejected (`NOT_RESUMABLE`)

### Scenario: The resumed run is one complete backend trace
- **Given** a run that paused and resumed
- **When** the resumed run finishes
- **Then** a single trace holds `Chat Input → Human In The Loop — {decision} → … → Chat Output`
- **And** the trace panel shows the gate + terminal output after a full page refresh, with no client-side persistence

### Scenario: A decision applied after timeout is rerouted, not lost
- **Given** a pending request that has a timeout policy
- **When** the decision arrives after the deadline
- **Then** `reroute_decision_on_timeout` resolves the effective decision deterministically

---

## 5. Architecture Decision Records

### ADR-001: The pause is a signal, not a failure

**Status**: Accepted

#### Context
A component needing human input must stop the graph without corrupting state or marking the run failed.

#### Decision
Model the pause as a dedicated exception (`GraphPausedException`) raised at a **layer boundary**, after the graph snapshots itself and writes a checkpoint. The build seam catches it and emits a **non-terminal** `human_input_required` event — the stream ends *without* `on_end`, so the run is "waiting", not "done".

#### Consequences
- **Benefits**: clean separation of "waiting" from "failed"; resumable by construction; default-off (no probe → no-op).
- **Trade-offs**: every layer boundary consults a pause probe (cheap; skipped entirely when no pause is pending).

### ADR-002: Resume is single-flight and rolls back

**Status**: Accepted

#### Context
A decision must apply exactly once even under duplicate clicks, retries, or concurrent callers.

#### Decision
Claim the job with an atomic `SUSPENDED → IN_PROGRESS` transition (`claim_suspended_for_resume`); on failure during resume, roll the status back so the decision is never silently consumed.

#### Consequences
- **Benefits**: exactly-once decisions; safe retries; no lost approvals.
- **Trade-offs**: a stale/duplicate resume returns `NOT_RESUMABLE` (handled by the UI).

### ADR-003: Resume ≠ re-run

**Status**: Accepted

#### Context
Re-executing completed vertices on resume would re-bill LLMs, re-fire tools, and could revive dead branches.

#### Decision
Restore built vertices from the checkpoint; re-run only the paused vertex's **non-input predecessors whose dropped output an unbuilt consumer will actually read** (`_rerun_non_input_predecessors` / `_unbuild_needed_dropped_producers`). A producer behind a still-built, round-tripped consumer is left alone — re-running it is wasted work and, for a side-effecting node (an Agent re-bills its LLM and re-emits its message), surfaces as duplicate outputs on every later resume. Inputs (e.g. Chat Input) are never re-executed.

Selecting the resume layer is not enough to hold that invariant. A resume restores `vertices_to_run` verbatim from the checkpoint, so vertices that came back **built** stay in the runnable pool; because `RunnableVerticesManager.is_vertex_runnable` never consulted `built`, the backward walk that looks for runnable predecessors when a successor is blocked (`find_runnable_predecessors_for_successor`) could hand one back to the build loop mid-resume. `Graph.is_vertex_runnable` therefore rejects a vertex that is still `built` **and** was restored from the checkpoint (`checkpoint_restored_built_ids`), excluding loop vertices, which legitimately re-run. Reading the live `built` flag rather than the checkpoint's is what keeps the gated node and the opaque-dropped producers — built at checkpoint time, deliberately un-built by the resume — eligible.

#### Consequences
- **Benefits**: no double billing, no duplicate side effects, deterministic continuation.
- **Trade-offs**: the resume re-runs a minimal predecessor set; the terminal output is produced fresh on resume (see ADR-004).
- **Symptom this prevents**: a re-executed Chat Input persists a second `User` message for the same turn, so the paused chat renders the user's question twice after the decision.

### ADR-004: The whole run is one durable backend trace (gate + Chat Output as spans)

**Status**: Accepted (2026-06-23)

#### Context
Originally, resumed runs lost trace data in the backend: the **Chat Output** span was missing and the **Human In The Loop** step existed only as a frontend-injected node persisted in `localStorage` — invisible to other devices/users and incomplete in the DB. Root cause: the resume path **never initialized tracing** (`trace_context_var` was unset, so post-pause vertices skipped `add_trace`), and the initial run traced under a throwaway uuid instead of the `run_id`, splitting the run into an orphan trace.

#### Decision
1. On resume, call `graph.initialize_run()` on the checkpoint's `run_id` so resumed vertices trace into the **same** trace as the pre-pause run, and restore `flow_name` for the trace title.
2. Pin the caller's `run_id` in `build_graph_from_data` **before** `initialize_run` (forwarded by `create_graph`), so the initial run traces into `graph_run_id` — not a fresh uuid.
3. Record the resolved gate as a real span via `TracingService.record_event_span` ("Human In The Loop — {action label}", e.g. Approve/Reject/Remove).
4. Remove the frontend `localStorage` persistence; `TraceDetailView` reads the gate + output from the backend trace and only synthesizes a gate for the live window, deduped by name.

#### Consequences
- **Benefits**: one complete trace (`Chat Input → gate → … → Chat Output`) durable across refresh, devices, and users; no client-side band-aid.
- **Trade-offs**: the resume re-initializes tracing once (a no-op when tracing is disabled).
- **Impact on Product**: the trace panel is now the source of truth for what a HITL run did.

### ADR-005: Terminal span flush race fixed

**Status**: Accepted

#### Context
Component spans are recorded asynchronously via a worker queue. At end-of-run the worker was cancelled while the **terminal** component's end event was still enqueued, dropping the Chat Output span in ~7 of 8 runs.

#### Decision
Drain the queue **inline** after cancelling the worker (`service.py::_stop`), and force-complete any started-but-unended span at flush (`native.py::_finalize_pending_spans`).

#### Consequences
- **Benefits**: no executed component is silently dropped from a trace.
- **Trade-offs**: a tiny synchronous drain at shutdown (bounded by queue size).

---

## 6. Technical Specification

### 6.1 Submit supersedes stale pauses
- `BackgroundExecutionService.submit` calls `supersede_suspended_runs(flow_id, user_id, session_id)` after `create_job` (so idempotent retries still return the existing job) and before enqueue: every `SUSPENDED` workflow job of the same flow + user **+ effective session** (`session_id` from the submit request, falling back to the flow id) is cancelled — in-process through `_cancel_suspended` (status `CANCELLED`, card stamped superseded, checkpoint deleted, pending cleared, `run_cancelled` event, bus closed), in scaled mode through `_backend.stop`. A stale run's session is read back off its persisted row (`job_metadata['request']['session_id']`, or the flat `job_metadata['session_id']` legacy rows carry) with the same flow-id fallback the runner uses for `thread_id`, so SUSPENDED jobs of OTHER sessions are left untouched and one flow can serve many concurrent callers ([#14599](https://github.com/langflow-ai/langflow/issues/14599)).
- The cancel is an **atomic claim**: `JobService.claim_suspended_for_cancel` (a conditional `UPDATE … WHERE status = SUSPENDED`, the mirror of `claim_suspended_for_resume`) flips the row, and cleanup runs only for rows this caller actually claimed. A resume that won the flip first keeps its run — supersede can never cancel a resumed job or destroy its checkpoint mid-flight. `stop_job` uses the same claim and falls through to the running-job STOP path when it loses.
- Before metadata is cleared, `mark_card_superseded` (`api/v2/hitl.py`) patches the persisted card's `human_input` content with `superseded: true` (skipping already-answered cards); `HumanInputCard` renders that state closed, so a reloaded history cannot re-offer a decision that would only 409.
- `DurableServeWorkflowHost.submit_background` mirrors it flow-scoped: `SqliteDurableJobStore.claim_suspended_for_cancel` first, then (only for claimed rows) STOP signal, task cancel, pending metadata cleared — a lost claim also means no stray STOP signal for the resumed continuation to consume.
- Frontend: the canvas badge auto-open is keyed by `request_id`, so the superseding run's new pause reopens a dismissed popover; the 5s pending poll converges every surface onto the single remaining pause.

### 6.2 Pause → suspend (initial run)
- The graph consults a pause probe at each layer boundary; on a pending pause it snapshots itself, writes a `checkpoint` row (always-writable schema), and raises `GraphPausedException`.
- The build seam (`api/build.py`) catches it, persists the human-input card to history, emits `human_input_required`, and ends the stream without `on_end`.
- The runner marks the job `SUSPENDED` (`services/background_execution/runner.py`).

### 6.3 Resume (single-flight)
- `POST /api/v2/workflows/{job_id}/resume` with `{request_id, decision}`.
- The route re-enforces `FlowAction.EXECUTE` (`ensure_resume_execute_permission`) before applying the decision — job ownership alone is not enough once shared access has been revoked.
- `claim_suspended_for_resume` performs the atomic status claim; failure → `NOT_RESUMABLE`.
- The answered card is stamped via `mark_card_answered` using the `card_message_id` snapshotted **before** the continuation is enqueued, and only when the card's `request_id` matches — so a first decision can never resolve a second pause's card.
- The gated vertex is located with `request_id_targets_vertex` (`lfx.run.hitl`), which accepts both the node shape and the nonce-suffixed tool-approval shape.
- `build_resumed_graph_and_get_order` (`api/build.py`):
  ```python
  graph = LfxGraph.resume_from_checkpoint(checkpoint, checkpoint_store=store)
  graph.flow_name = graph.flow_name or flow_name
  await graph.initialize_run()                      # re-init tracing on the original run_id
  decision = reroute_decision_on_timeout(pending, resume["decision"])
  graph.human_input_decisions = {resume["request_id"]: decision}
  action_id = str((decision or {}).get("action_id", ""))
  # HITL actions are user-defined (Approve/Reject/Remove/...), so label the span with the chosen
  # action's button label (from the pending options), not a hardcoded approve/reject binary.
  gate_label = _hitl_gate_label(action_id, (pending or {}).get("options"))
  graph.tracing_service.record_event_span(
      span_id=f"hitl-{resume['request_id']}",
      name=f"Human In The Loop — {gate_label}",
      outputs={"decision": action_id},
  )
  # restore built vertices; re-run only non-input predecessors of the gated vertex
  ```

### 6.4 Trace continuity (initial run)
- `build_graph_from_data` (`api/utils/flow_utils.py`) pins the run id before tracing starts:
  ```python
  if (caller_run_id := kwargs.get("run_id")) is not None:
      graph.set_run_id(caller_run_id)
  await graph.initialize_run()
  ```
- `create_graph` forwards `run_id=str(job_id) if job_id is not None else run_id` to both `build_graph_from_data` and `build_graph_from_db`.

### 6.5 Gate span recording
- `TracingService.record_event_span(span_id, name, outputs, span_type="chain")` writes a standalone start+end span into the active trace for every ready tracer; deactivated/contextless calls are no-ops.

### 6.6 Frontend (reads backend spans)
- `useGetTraceQuery` → `GET /api/v1/monitor/traces/{id}` returns the span tree.
- `TraceDetailView.tsx`: renders the backend gate span; `backendHasGate` dedup prevents a duplicate synthetic node; `executedOutputSpans` bridges Chat Output from live `flowPool` only during the resume window (deduped by name).
- `hitlStore.ts`: in-memory only (`pending` slot); `localStorage`/`persist` removed.
- Answered-card propagation: the chat's session cache is a subscription-only query (`staleTime: Infinity`, fed by `setQueryData`), so `invalidateQueries` cannot refresh it and it re-hydrates from the backend only while empty. A decision taken on another surface — canvas badge, trace bar, another tab — is therefore adopted on load by `withAnsweredHumanInputCards` (`human-input-card.ts`), which stamps `submitted_action` from the backend copy onto cards the cache still shows as open. Without it an answered pause keeps rendering its buttons until a full page reload.

### 6.7 lfx serve durable mode (LE-1695)

Bare `lfx serve` gains the same background + HITL contract without a database, backed by the single-node SQLite substrate:

- **Substrate** (`lfx/services/durable/`): `SqliteDurableJobStore` (job rows, gap-free per-job event log, control signals, atomic `claim_suspended_for_resume`) and `SqliteCheckpointStore` (the existing `CheckpointStore` ABC on the same DB file). Stdlib SQLite, WAL, async via `asyncio.to_thread`; one crash-safe file per deployment.
- **Host** (`lfx/cli/serve_durable.py`): `DurableServeWorkflowHost` extends `ServeWorkflowHost` with `supports_background = True`; `submit_background` creates the job row and spawns the runner; `get_job_status` rebuilds a completed run's `WorkflowExecutionResponse` from stored `output_events`; `stop_job` records a durable STOP signal and cancels the local task.
- **Runner** (`_drive`): runs `graph.process()` with checkpointing attached; `GraphPausedException` → job `SUSPENDED` + pending request persisted in job metadata; completion → terminal `ComponentOutput`s persisted as the job result; failure → `FAILED` with the error recorded.
- **Resume**: `POST /api/v2/workflows/{job_id}/resume` (same request/response contract as the backend, incl. 404/409 semantics and the 422 `INVALID_DECISION` guard against actions outside `allowed_decisions`) claims single-flight, restores from the checkpoint — in-process or after a restart — re-applies the caller identity, injects the decision, un-builds the gated vertex plus its opaque-dropped producers, and re-drives. Agent pause blobs persist in the same SQLite file via `set_default_checkpoint_store`, so tool-approval interrupts also survive a restart. `GET /api/v2/workflows/{job_id}/pending` exposes the pending request.
- **Events (SSE)**: `GET /api/v2/workflows/{job_id}/events` re-attaches to a background run — the `links.events` URL advertised on every job response. It replays the durable event log after `Last-Event-ID` (each frame keyed by its `seq`), then tails until the run ends or suspends; a HITL pause replays and closes the stream, so the client reconnects with `Last-Event-ID` after resuming. Unknown job → 404.
- **Opt-in**: `LFX_SERVE_DURABLE_DB=<path>`. Unset → serve stays stateless and `mode: background` keeps its 422; workers in multi-worker mode inherit the env var and share the DB file (a stop cancels the run only in the worker executing it).
- **Proof**: `src/lfx/tests/unit/cli/test_serve_durable.py` — background echo completes with the sync-shaped response; a HITL flow suspends, exposes its pending request, resumes only the approved branch, and resumes to completion on a fresh app instance over the same DB file (restart equivalence). The `{job_id}/events` SSE stream replays a completed run, surfaces the `human_input_request` on a suspend, and — reconnected with `Last-Event-ID` after a resume — delivers only the post-resume `human_input_decision` + `end` frames.

### 6.8 Component map (quick index)

| Concern | Where | Key symbols |
|---------|-------|-------------|
| Pause signal + checkpoint | `lfx/graph/graph/base.py`, `lfx/graph/checkpoint/` | `GraphPausedException`, `resume_from_checkpoint`, `set_run_id` |
| Build seam + resume rebuild | `api/build.py` | `build_resumed_graph_and_get_order`, `_rerun_non_input_predecessors` |
| Resume scheduling guard | `lfx/graph/graph/base.py` | `is_vertex_runnable`, `checkpoint_restored_built_ids` |
| Run-id pinning | `api/utils/flow_utils.py` | `build_graph_from_data` |
| Durable job + single-flight | `services/background_execution/{runner,service}.py` | `JobStatus.SUSPENDED`, `claim_suspended_for_resume` |
| lfx serve durable substrate | `lfx/services/durable/`, `lfx/cli/serve_durable.py` | `SqliteDurableJobStore`, `SqliteCheckpointStore`, `DurableServeWorkflowHost` |
| Tracing | `services/tracing/{native,service}.py` | `record_event_span`, `_finalize_pending_spans`, `_stop` drain |
| Trace UI | `TraceComponent/TraceDetailView.tsx` | `backendHasGate`, `executedOutputSpans` |
| Live HITL slot | `stores/hitlStore.ts` | `pending` (in-memory) |

---

## 7. Observability

- **Trace panel** (Flow Activity): the run renders as a span tree — `Chat Input → tools → Agent → Chat Output` — and, for a gated run, a `Human In The Loop — {decision}` span. Read via `GET /api/v1/monitor/traces/{id}`.
- **Job status**: `SUSPENDED` / `IN_PROGRESS` / `COMPLETED` / `FAILED` exposed through the workflows API; pending requests via `GET /api/v2/workflows/pending?flow_id=`.
- **Validated end-to-end** (2026-06-23): a driven pause→approve→resume yields **one** trace of 21 spans (`Chat Input → Human In The Loop — Approved → Calculator → URL → Agent → fetch_content → … → Chat Output`), confirmed in the DB, in the `/monitor/traces/{id}` payload, and in the trace panel after a page refresh.

---

## 8. Deployment & Rollback

- **Default-off**: no component requests a pause → no checkpointing, no extra spans, no status churn. Safe to ship.
- **Schema**: the `checkpoint` and `span`/`trace` tables back the durability + observability; no destructive migrations are part of this feature.
- **Run modes**: works under the durable background path (`POST /api/v2/workflows`, `mode: background`) with resume via `POST /api/v2/workflows/{job_id}/resume`. On bare `lfx serve`, the same contract is opt-in via `LFX_SERVE_DURABLE_DB=<path>` (SQLite substrate, no database service); unset keeps serve stateless.
- **Rollback**: reverting the trace-continuity change restores the prior behavior (Chat Output/gate absent from resumed backend traces) but does not affect pause/resume correctness; reverting the pause/resume layer disables HITL entirely (flows run straight through).

---

## 9. Architecture Diagrams

### 9.1 Context Diagram (Level 1)

```mermaid
C4Context
  title System Context — Durable HITL
  Person(user, "User", "Approves/rejects a gated step")
  System(langflow, "Langflow", "Flow execution platform")
  SystemDb(db, "Database", "checkpoint, job status, trace/span")
  Rel(user, langflow, "Runs flow; answers Approve/Reject")
  Rel(langflow, db, "Persists checkpoint + status + spans")
  Rel(langflow, user, "Surfaces card / trace gate")
```

### 9.2 Sequence — pause → suspend → resume

```mermaid
sequenceDiagram
  participant U as User
  participant API as Workflows API
  participant G as Graph Engine
  participant DB as Database
  participant T as Tracing

  U->>API: POST /v2/workflows (background)
  API->>G: build + run (run_id pinned)
  G->>T: trace Chat Input … Agent
  G->>DB: write checkpoint (layer boundary)
  G-->>API: GraphPausedException → human_input_required
  API->>DB: job SUSPENDED
  API-->>U: Approve / Reject card
  U->>API: POST /v2/workflows/{job}/resume {decision}
  API->>DB: claim SUSPENDED→IN_PROGRESS (single-flight)
  API->>G: resume_from_checkpoint + initialize_run(run_id)
  G->>T: record "Human In The Loop — {decision}" span
  G->>T: trace re-run predecessors … Chat Output
  G->>DB: flush spans (one trace) + job COMPLETED
  API-->>U: result; trace panel shows full run
```

### 9.3 Trace continuity (why one trace)

```mermaid
graph TB
    A[POST /v2/workflows] --> B[create_graph]
    B --> C[build_graph_from_data]
    C --> D{run_id pinned?}
    D -->|yes set_run_id job_id| E[initialize_run → trace = graph_run_id]
    D -->|no - old bug| F[fresh uuid → orphan trace]
    E --> G[pause → checkpoint stores run_id]
    G --> H[resume_from_checkpoint set_run_id run_id]
    H --> I[initialize_run → SAME trace]
    I --> J[record gate span + Chat Output]
    J --> K[(One complete trace)]
```

### 9.4 Frontend read path (no localStorage)

```mermaid
graph LR
    P[Trace panel] --> Q[useGetTraceQuery]
    Q --> R[GET /api/v1/monitor/traces/id]
    R --> S[span tree incl. gate + Chat Output]
    S --> T[TraceDetailView renders backend spans]
    T --> U{backendHasGate?}
    U -->|yes| V[use backend gate - no synth]
    U -->|live window| W[synth gate / bridge Chat Output from flowPool]
```