will/swarm-zap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
d08d8fe661793d4344d9347a636255b207102e1a
swarm-zap/memory/2026-03-13.md

14 KiB
Raw Blame History

2026-03-13

Subagent reliability investigation

  • Fresh implementation subagent launch for subagent/ACP reliability failed immediately before doing any task work.
  • Failure mode: delegated run was spawned with model glm-5, which resolved to provider model zai/glm-5.
  • Current installed agent auth profile keys inspected in agent stores include openai-codex:default, litellm:default, and github-copilot:github.
  • Will clarified on 2026-03-13 that Z.AI auth does exist in the environment, but the account is not entitled for glm-5.
  • Verified by inspecting agent auth profile keys under:
    • /home/openclaw/.openclaw/agents/*/agent/auth-profiles.json
  • Relevant OpenClaw docs confirm:
    • subagent spawns inherit caller model when sessions_spawn.model is omitted
    • provider/model auth errors like No API key found for provider "zai" occur when a provider model is selected without matching auth
    • multi-agent auth is per-agent via ~/.openclaw/agents/<agentId>/agent/auth-profiles.json
  • Conclusion: the immediate failure was caused by an incorrect explicit model selection in the spawn request, not by missing auth propagation between agents.
  • Corrective action: retry fresh delegation with litellm/glm-5 (the intended medium-tier routed model for delegated implementation work in this setup).
  • Will explicitly requested on 2026-03-13 to use gpt-5.4 for subagents for now while debugging delegation reliability.
  • New evidence from the corrected run: ~/.openclaw/agents/main/sessions/1615a980-cf92-4d5e-845a-a2abe77c0418.jsonl shows repeated assistant stopReason:"error" entries with 429 ... GLM-5 not included in current subscription plan, but ~/.openclaw/subagents/runs.json recorded run 776a8b51-6fdc-448e-83bc-55418814a05b as outcome.status: "ok" and frozenResultText: null.
  • That separates ACP/runtime choice problems from a generic subagent completion/reporting bug: a terminal assistant error can still be persisted/announced as success with no useful result.
  • Implemented upstream fix on branch external/openclaw-upstream@fix/subagent-wait-error-outcome:
    • added assistant terminal-outcome helper so empty-content assistant errors still yield usable terminal text
    • subagent registry now downgrades agent.wait => ok to error when the child session's terminal assistant message is actually an error
    • subagent announce flow now reports terminal assistant errors as failed outcomes instead of successful (no output) completions
  • Targeted validation passed:
    • pnpm -C /home/openclaw/.openclaw/workspace/external/openclaw-upstream test -- --run src/agents/tools/sessions-helpers.terminal-text.test.ts src/agents/subagent-registry.persistence.test.ts src/gateway/server-methods/server-methods.test.ts
    • result: 50 tests passed across 3 files
  • Real success-path verification later passed on gpt-5.4 with run 23750d80-b481-4f50-b219-cc9245be405f and final child result SUCCESS-PROBE-OK.
  • Real failure-path verification later also passed on valid gpt-5.4 by intentionally triggering a context_length_exceeded provider error with a token-dense oversized task payload.
    • child run: b50cb91f-6219-44f7-9d2f-a1264ac7ceaf
    • child session: agent:main:subagent:4c0dd686-cd2e-4cba-b80b-2fbf309a4594
    • child transcript: ~/.openclaw/agents/main/sessions/f114b831-000b-4070-a539-85c68d2b7057.jsonl
    • transcript terminal assistant entry recorded provider:"openai-codex", model:"gpt-5.4", stopReason:"error", errorMessage:"Codex error: {...context_length_exceeded...}"
    • matching ~/.openclaw/subagents/runs.json now correctly stored:
      • outcome.status: "error"
      • outcome.error: "Codex error: {...context_length_exceeded...}"
      • endedReason: "subagent-error"
      • frozenResultText: "Codex error: {...context_length_exceeded...}"
  • Important remaining nuance from the live repro: raw gateway agent.wait for that same failed child returned status:"ok" with only endedAt even though the child transcript terminal assistant message had stopReason:"error".
  • Follow-up code inspection on 2026-03-13 showed this is an upstream bug, not an intentional agent.wait layering choice:
    • embedded subscribe lifecycle already emits phase:"error" for terminal assistant/provider failures
    • but src/commands/agent.ts had a fallback lifecycle emitter that still sent phase:"end" whenever no inner lifecycle callback was observed, even if the resolved run result carried meta.stopReason:"error"
    • waitForAgentJob gives lifecycle errors a retry grace window, so that fallback end could overwrite the terminal failure and make agent.wait resolve ok
  • Implemented focused upstream follow-up on branch fix/subagent-wait-error-outcome:
    • src/commands/agent.ts now emits lifecycle phase:"error" with extracted terminal error text when a resolved run stops with meta.stopReason:"error" and no inner lifecycle callback fired
    • src/gateway/server-methods/agent-wait-dedupe.ts now also maps completed agent dedupe payloads with result.meta.stopReason:"error" to status:"error" and aborted:true to status:"timeout"
  • Targeted validation passed:
    • pnpm -C /home/openclaw/.openclaw/workspace/external/openclaw-upstream test -- --run src/commands/agent.test.ts src/gateway/server-methods/agent-wait-dedupe.test.ts src/gateway/server-methods/server-methods.test.ts
    • result: 81 tests passed across 3 files
  • Live runtime verification was re-run later on 2026-03-13 and showed the current agent.wait follow-up fix still does not hold on the live direct gateway path.
    • first temp-gateway sanity run via GatewayClient against loopback port 18901 on a persisted gpt-5.4 session returned status:"error", but only because that temp runtime reported FailoverError: Unknown model: openai-codex/gpt-5.4; useful as transport sanity, not canonical semantics proof
    • stale-dist temp gateway repro on default model (gpt-5.3-codex) already showed the mismatch:
      • session key: agent:main:subagent:agent-wait-gpt53-live-1773427893572
      • run id: gwc-live-agent-wait-gpt53-1773427893583
      • agent.wait: {"runId":"gwc-live-agent-wait-gpt53-1773427893583","status":"ok","endedAt":1773427896100}
      • last assistant still recorded stopReason:"error" with context_length_exceeded
    • decisive live source-gateway repro used a fresh source-run gateway on port 18902 launched with:
      • OPENCLAW_SKIP_CHANNELS=1 CLAWDBOT_SKIP_CHANNELS=1 pnpm exec tsx src/index.ts gateway run --port 18902 --bind loopback --auth none --allow-unconfigured
      • gateway log confirmed default model openai-codex/gpt-5.3-codex
      • session key: agent:main:subagent:agent-wait-gpt53-live-source-1773427981586
      • run id: gwc-live-agent-wait-gpt53-source-1773427981614
      • payload chars: 880150
      • start: {"runId":"gwc-live-agent-wait-gpt53-source-1773427981614","status":"accepted","acceptedAt":1773427981959}
      • agent.wait: {"runId":"gwc-live-agent-wait-gpt53-source-1773427981614","status":"ok","endedAt":1773427984243}
      • same session's terminal assistant message still recorded:
        • provider:"openai-codex"
        • model:"gpt-5.3-codex"
        • stopReason:"error"
        • errorMessage:"Codex error: {\"type\":\"error\",\"error\":{\"type\":\"invalid_request_error\",\"code\":\"context_length_exceeded\",\"message\":\"Your input exceeds the context window of this model. Please adjust your input and try again.\",\"param\":\"input\"},\"sequence_number\":2}"
  • Fast source inspection after that live repro points to the most likely remaining gap:
    • src/commands/agent.ts only emits the new corrective lifecycle phase:"error" when !lifecycleEnded
    • lifecycleEnded becomes true as soon as any inner lifecycle callback reports phase:"end" or phase:"error"
    • src/gateway/server-methods/agent-job.ts still treats lifecycle phase:"end" as terminal status:"ok"
    • so the likeliest still-open live bug is an inner lifecycle emitter marking terminal assistant/provider failures as end early enough that agent.wait resolves ok before the dedupe/result-meta rescue path matters
  • Net status at end of this pass:
    • subagent persistence/announcement fix: live-verified
    • raw agent.wait follow-up fix: tests passed, but live source-gateway repro still failed; do not mark this closed
  • Final focused live-fix pass on 2026-03-13 closed the remaining raw agent.wait bug.
    • root cause: the live direct gateway path could receive agent_end carrying a terminal assistant error without a preceding message_end, leaving stale/empty assistant state and still emitting lifecycle phase:"end"
    • final upstream fix taught embedded subscribe lifecycle handling to recover the terminal assistant from agent_end.messages / session transcript and emit lifecycle phase:"error", and taught the gateway agent RPC handler to derive terminal status from observed lifecycle + final result metadata instead of blindly caching ok
    • final targeted validation passed:
      • pnpm -C /home/openclaw/.openclaw/workspace/external/openclaw-upstream test -- --run src/agents/pi-embedded-subscribe.handlers.lifecycle.test.ts src/agents/pi-embedded-subscribe.subscribe-embedded-pi-session.subscribeembeddedpisession.test.ts src/commands/agent.test.ts src/gateway/server-methods/agent-wait-dedupe.test.ts src/gateway/server-methods/server-methods.test.ts
      • result: 108 tests passed across 5 files
    • decisive live source-gateway repro after the final fix:
      • gateway: source-run on port 18903
      • run id: gwc-live-agent-wait-gpt53-source-fixed2-1773429512008
      • final agent response returned finalStatus:"error"
      • matching agent.wait returned status:"error" with the same context-window error text
  • Net status now:
    • subagent persistence/announcement fix: live-verified ✅
    • raw agent.wait semantics fix: live-verified ✅
  • Side note: unrelated dirty /subagents log UX changes in external/openclaw-upstream regression-passed src/auto-reply/reply/commands.test.ts (44 tests) but were intentionally left out-of-scope for this focused reliability pass.

ACP Claude/Codex follow-up (post-agent.wait fix)

  • Historical deferred task task-20260304-211216-acp-claude-codex still referenced old failures Claude: acpx exited with code 1 and Codex: acpx exited with code 5, but those exact crashes were not reproduced in the latest focused pass.
  • Current host state check:
    • claude installed: /home/linuxbrew/.linuxbrew/bin/claude (2.1.63)
    • codex installed: /home/linuxbrew/.linuxbrew/bin/codex (0.107.0)
    • no global acpx on PATH, but bundled plugin-local runtime exists at ~/.local/share/pnpm/.../openclaw/extensions/acpx/node_modules/.bin/acpx
    • current ~/.openclaw/openclaw.json only showed plugins.entries.telegram.enabled=true; no explicit acp block / acpx plugin entry was present, so the smallest reliable repro path used the bundled acpx directly rather than a full OpenClaw ACP session
  • Live direct bundled-acpx repro results:
    • Codex command:
      • .../acpx --format json --json-strict --timeout 15 codex exec 'reply with OK only'
      • result: clean JSON-RPC/session stream ended with agent_message_chunk: "OK", id:2 result:{stopReason:"end_turn"}, process exit=0
    • Claude command:
      • .../acpx --format json --json-strict --timeout 20 claude exec 'reply with OK only'
      • stdout included top-level JSON-RPC errors:
        • {"jsonrpc":"2.0","id":2,"error":{"code":-32000,"message":"Authentication required"}}
        • {"jsonrpc":"2.0","id":null,"error":{"code":-32000,"message":"Authentication required"}}
      • process still exited 0
  • Source-level finding in external/openclaw-upstream/extensions/acpx/src/runtime-internals/events.ts:
    • prompt parsing handled typed {type:"error"} lines but dropped top-level JSON-RPC error responses
    • that meant runtime.runTurn() could treat a Claude auth failure as success (done) when the agent emitted JSON-RPC errors yet exited cleanly
  • Implemented focused upstream fix on branch fix/subagent-wait-error-outcome:
    • extensions/acpx/src/runtime-internals/events.ts
      • toAcpxErrorEvent() now also recognizes top-level JSON-RPC error responses via parseControlJsonError()
      • parsePromptEventLine() now emits ACP runtime type:"error" events for that shape instead of dropping it
    • added regression coverage:
      • extensions/acpx/src/runtime-internals/events.test.ts
      • extensions/acpx/src/runtime-internals/test-fixtures.ts
      • extensions/acpx/src/runtime.test.ts
  • Targeted validation passed:
    • cd /home/openclaw/.openclaw/workspace/external/openclaw-upstream && pnpm exec vitest run extensions/acpx/src/runtime-internals/events.test.ts extensions/acpx/src/runtime.test.ts extensions/acpx/src/runtime-internals/control-errors.test.ts
    • result: 22 tests passed across 3 files
  • Net status after this pass:
    • old acpx exited with code 1/5 reports remain historical evidence only
    • Codex ACP direct runtime path works today
    • Claude ACP direct runtime path currently fails for auth, and OpenClaw had a real bug in how the bundled acpx runtime parsed that failure shape
    • remaining follow-up is end-to-end OpenClaw ACP-path validation once ACP is explicitly configured here (or if a fresh exit-code repro appears)
  • Will also explicitly requested that zap keep a light eye on active subagents and check whether they look stuck instead of assuming they are fine until completion.
  • Will explicitly reinforced on 2026-03-13 that once planning is done, zap should use subagents ASAP and start implementation in a fresh session rather than continuing to implement inside the long-lived main chat.
  • Will explicitly asked on 2026-03-13 for more frequent checks on active subagent runs; zap should inspect/steer sooner instead of waiting for long silent stretches.