feat(lifecycle): stuck detection + heartbeat lifecycle + SDK tool blocklist
Replaces the two overlapping old mechanisms (30-min setTimeout kill in
container-runner, 10-min heartbeat STALE_THRESHOLD reset in host-sweep)
with message-scoped stuck detection anchored to the processing_ack claim
age + an absolute 30-min ceiling that extends for long-declared Bash
tools.
Old model problems:
- IDLE_TIMEOUT setTimeout fired on plain wall-clock time; slow-but-alive
agents got killed at 30min regardless of activity
- 10-min STALE_THRESHOLD in the sweep was unreliable — the heartbeat is
only touched on SDK events, so legitimate silent tool work (sleep 30,
long WebFetch, npm install) looked identical to a hung container
- Two overlapping sources of truth for "when to let go of a container"
New model:
- Host sweep is the single source of truth.
- Container exposes a new `container_state` single-row table in outbound.db
(schema added; container writes, host reads). PreToolUse hook writes
current_tool + tool_declared_timeout_ms (read from Bash's tool_input);
PostToolUse / PostToolUseFailure clear it.
- Sweep decides with a pure helper `decideStuckAction`:
* absolute ceiling — kill if heartbeat age > max(30min, bash_timeout)
* per-claim stuck — kill if any processing_ack row has claim_age >
max(60s, bash_timeout) AND heartbeat hasn't been touched since claim
* otherwise ok
Kill paths reset leftover processing rows with exponential backoff,
reusing the existing retry machinery.
Tool blocklist expanded:
- AskUserQuestion (SDK placeholder; we have mcp__nanoclaw__ask_user_question)
- EnterPlanMode, ExitPlanMode, EnterWorktree, ExitWorktree (Claude Code UI
affordances; would hang in headless containers)
PreToolUse hook is also defense-in-depth: if a disallowed tool name slips
through, it returns `{ decision: 'block' }` so the agent sees a clear
error instead of appearing stuck.
Removed:
- container-runner.ts: IDLE_TIMEOUT setTimeout, resetIdle callback on
activeContainers entry, resetContainerIdleTimer export.
- delivery.ts: the resetContainerIdleTimer call on successful delivery.
- poll-loop.ts: IDLE_END_MS + its setInterval. Keeping the query open is
cheaper than close+reopen (no cold prompt cache). Liveness is now a
host-side concern.
- host-sweep.ts: 10-min STALE_THRESHOLD_MS + getStuckProcessingIds in the
stale-detection path (still exported for kill reset).
Tests:
- src/host-sweep.test.ts — 9 tests for decideStuckAction covering: fresh
heartbeat, absolute ceiling, absent heartbeat, Bash-timeout extension
(both ceiling and per-claim), claim age below tolerance, heartbeat
touched after claim, unparseable timestamps.
Ref: docs/v1-vs-v2/ACTION-ITEMS.md items 9, 6a, 10.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -64,10 +64,58 @@ export function getOutboundDb(): Database {
|
||||
if (!cols.has('updated_at')) {
|
||||
_outbound.exec(`ALTER TABLE session_state ADD COLUMN updated_at TEXT NOT NULL DEFAULT ''`);
|
||||
}
|
||||
// container_state: tracks the current tool in flight (if any) so the host
|
||||
// sweep can widen its stuck tolerance when Bash is running with a user-
|
||||
// declared long timeout. Forward-compat for older outbound.db files.
|
||||
_outbound.exec(`
|
||||
CREATE TABLE IF NOT EXISTS container_state (
|
||||
id INTEGER PRIMARY KEY CHECK (id = 1),
|
||||
current_tool TEXT,
|
||||
tool_declared_timeout_ms INTEGER,
|
||||
tool_started_at TEXT,
|
||||
updated_at TEXT NOT NULL
|
||||
);
|
||||
`);
|
||||
}
|
||||
return _outbound;
|
||||
}
|
||||
|
||||
/**
|
||||
* Record that a tool is starting. `declaredTimeoutMs` is the tool's own
|
||||
* timeout hint when one is available (Bash exposes it in the tool_use input);
|
||||
* omit for tools with no declared timeout.
|
||||
*/
|
||||
export function setContainerToolInFlight(tool: string, declaredTimeoutMs: number | null): void {
|
||||
const now = new Date().toISOString();
|
||||
getOutboundDb()
|
||||
.prepare(
|
||||
`INSERT INTO container_state (id, current_tool, tool_declared_timeout_ms, tool_started_at, updated_at)
|
||||
VALUES (1, ?, ?, ?, ?)
|
||||
ON CONFLICT(id) DO UPDATE SET
|
||||
current_tool = excluded.current_tool,
|
||||
tool_declared_timeout_ms = excluded.tool_declared_timeout_ms,
|
||||
tool_started_at = excluded.tool_started_at,
|
||||
updated_at = excluded.updated_at`,
|
||||
)
|
||||
.run(tool, declaredTimeoutMs, now, now);
|
||||
}
|
||||
|
||||
/** Clear the in-flight tool — called on PostToolUse / PostToolUseFailure. */
|
||||
export function clearContainerToolInFlight(): void {
|
||||
const now = new Date().toISOString();
|
||||
getOutboundDb()
|
||||
.prepare(
|
||||
`INSERT INTO container_state (id, current_tool, tool_declared_timeout_ms, tool_started_at, updated_at)
|
||||
VALUES (1, NULL, NULL, NULL, ?)
|
||||
ON CONFLICT(id) DO UPDATE SET
|
||||
current_tool = NULL,
|
||||
tool_declared_timeout_ms = NULL,
|
||||
tool_started_at = NULL,
|
||||
updated_at = excluded.updated_at`,
|
||||
)
|
||||
.run(now);
|
||||
}
|
||||
|
||||
/**
|
||||
* Touch the heartbeat file — replaces the old touchProcessing() DB writes.
|
||||
* The host checks this file's mtime for stale container detection.
|
||||
@@ -157,6 +205,13 @@ export function initTestSessionDb(): { inbound: Database; outbound: Database } {
|
||||
value TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL
|
||||
);
|
||||
CREATE TABLE container_state (
|
||||
id INTEGER PRIMARY KEY CHECK (id = 1),
|
||||
current_tool TEXT,
|
||||
tool_declared_timeout_ms INTEGER,
|
||||
tool_started_at TEXT,
|
||||
updated_at TEXT NOT NULL
|
||||
);
|
||||
`);
|
||||
|
||||
return { inbound: _inbound, outbound: _outbound };
|
||||
|
||||
Reference in New Issue
Block a user