diff --git a/README.md b/README.md index 769191f..6a6a67f 100644 --- a/README.md +++ b/README.md @@ -363,6 +363,19 @@ backends: `pi_embedded` is intended for canary migration cohorts. In spike mode (`no_tools_mode: true`), Flynn keeps tool-oriented turns on native and only routes plain-text turns to Pi. +To evaluate canary performance from audit logs, run: + +```bash +pnpm audit:backend-canary \ + --audit ~/.local/share/flynn/audit.log \ + --backend pi_embedded \ + --baseline native \ + --session telegram:8367012007 \ + --format markdown +``` + +Phase-2 evaluation checklist and decision template: `docs/plans/pi_embedded_evaluation.md`. + When `args` is non-empty: - use `{prompt}` in an argument to inject the full generated prompt directly into argv. - if `{prompt}` is not present, Flynn appends backend-specific prompt args. diff --git a/docs/api/PROTOCOL.md b/docs/api/PROTOCOL.md index 7a6da1e..b25f2cb 100644 --- a/docs/api/PROTOCOL.md +++ b/docs/api/PROTOCOL.md @@ -37,6 +37,7 @@ The gateway serialises agent work **per session**, not per WebSocket connection: - Lane policy is configurable (`collect`, `followup`, `steer`, `steer_backlog`, `interrupt`) with per-channel and per-session overrides. - Session-local overrides can be managed at runtime via `agent.send` commands: `/queue`, `/queue set ...`, `/queue reset`. - Backend selection for a turn is server-side (`native` by default, optional external backends per config: `claude_code`, `opencode`, `codex`, `gemini`, `pi_embedded`) and does not change JSON-RPC method signatures. +- Backend routing and fallback outcomes are emitted to audit logs (`backend.route`, `backend.success`, `backend.fallback`) for rollout evaluation; this telemetry is outside JSON-RPC response payloads. This is implemented via a per-lane queue (`LaneQueue`) in the gateway server, and used by `agent.send` and `agent.cancel`. diff --git a/docs/architecture/AGENT_DIAGRAM.md b/docs/architecture/AGENT_DIAGRAM.md index 31f437b..f5e4a17 100644 --- a/docs/architecture/AGENT_DIAGRAM.md +++ b/docs/architecture/AGENT_DIAGRAM.md @@ -139,6 +139,7 @@ Key files: - External backend adapters: `src/backends/external.ts`, `src/backends/piEmbedded.ts` - Model routing: `src/models/router.ts` - Tool policy + execution: `src/tools/policy.ts`, `src/tools/executor.ts` +- Canary backend telemetry summarization (offline evaluation): `src/audit/backendCanarySummary.ts`, `scripts/summarize-backend-canary.ts` ## Component Graph (Agent-Safety Boundary) diff --git a/docs/architecture/GATEWAY_SESSIONS_AND_QUEUE.md b/docs/architecture/GATEWAY_SESSIONS_AND_QUEUE.md index 874a92d..514223e 100644 --- a/docs/architecture/GATEWAY_SESSIONS_AND_QUEUE.md +++ b/docs/architecture/GATEWAY_SESSIONS_AND_QUEUE.md @@ -11,6 +11,7 @@ If you only want the protocol surface, see `docs/api/PROTOCOL.md`. - Agent work is queued per `sessionId` (FIFO), not per connection. - Sessions persist in SQLite via `SessionManager` even if clients disconnect. - Once dequeued, message routing may execute the native orchestrator path or an optional external backend path (`claude_code`, `opencode`, `codex`, `gemini`, `pi_embedded`) depending on agent/backend config. +- Backend routing outcomes are auditable via `backend.route` / `backend.success` / `backend.fallback`, which enables offline canary evaluation without changing gateway protocol methods. ## Component Map diff --git a/docs/plans/artifacts/.gitkeep b/docs/plans/artifacts/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/docs/plans/pi_embedded_evaluation.md b/docs/plans/pi_embedded_evaluation.md new file mode 100644 index 0000000..8499a32 --- /dev/null +++ b/docs/plans/pi_embedded_evaluation.md @@ -0,0 +1,114 @@ +# Pi Embedded Canary Evaluation (Phase 2) + +Status: in progress +Owner: Flynn maintainers +Started: 2026-02-24 + +## Goal + +Close the canary spike with a formal, repeatable evaluation and an explicit rollout decision (`expand`, `hold`, or `rollback`). + +## Scope + +- Target backend: `pi_embedded` +- Target cohort (current): `telegram:8367012007` +- Baseline backend: `native` +- Data source: audit events (`backend.route`, `backend.success`, `backend.fallback`, `session.message`) + +## Pass/Fail Gate + +Use the same thresholds for every evaluation window. + +| Metric | Gate | +| --- | --- | +| Completion rate delta (target - baseline) | >= -2.00pp | +| P50 latency delta (target - baseline) | <= +250ms | +| P95 latency delta (target - baseline) | <= +700ms | +| External fallback rate (`pi_embedded`) | <= 5.00% | +| Guardrail escapes | 0 unresolved | + +Notes: +- Completion rate and latency are computed from route-to-assistant turn timings. +- Fallback rate is computed from `backend.success` + `backend.fallback` attempt outcomes. +- Guardrail escapes are reviewed from `backend.route.source == forced_native_guard` + operator incident review. + +## How To Run + +Run a canary summary for the current cohort: + +```bash +pnpm audit:backend-canary \ + --audit ~/.local/share/flynn/audit.log \ + --backend pi_embedded \ + --baseline native \ + --session telegram:8367012007 \ + --format markdown +``` + +Run with gate evaluation and emit JSON artifact: + +```bash +pnpm audit:backend-canary \ + --audit ~/.local/share/flynn/audit.log \ + --backend pi_embedded \ + --baseline native \ + --session telegram:8367012007 \ + --format json \ + --out docs/plans/artifacts/pi_embedded_eval_latest.json \ + --gate-max-completion-drop-pp 2 \ + --gate-max-p50-latency-increase-ms 250 \ + --gate-max-p95-latency-increase-ms 700 \ + --gate-max-fallback-rate-pct 5 +``` + +## Evaluation Log + +### Window A + +- Dates: _TBD_ +- Route volume: _TBD_ +- Summary artifact: _TBD_ + +| Check | Result | Notes | +| --- | --- | --- | +| Completion rate delta | _TBD_ | | +| P50 latency delta | _TBD_ | | +| P95 latency delta | _TBD_ | | +| Fallback rate | _TBD_ | | +| Guardrail escapes | _TBD_ | | + +### Window B + +- Dates: _TBD_ +- Route volume: _TBD_ +- Summary artifact: _TBD_ + +| Check | Result | Notes | +| --- | --- | --- | +| Completion rate delta | _TBD_ | | +| P50 latency delta | _TBD_ | | +| P95 latency delta | _TBD_ | | +| Fallback rate | _TBD_ | | +| Guardrail escapes | _TBD_ | | + +## Tool Compatibility Findings + +Track all tool-adjacent/risky prompts that were force-routed to native (`no_tools_mode`) and any misses. + +| Class | Observed behavior | Action | +| --- | --- | --- | +| Tool-adjacent prompts | _TBD_ | | +| Capability-query prompts | _TBD_ | | +| Attachments-present turns | _TBD_ | | + +## Decision Record + +- Decision date: _TBD_ +- Decision: _expand | hold | rollback_ +- Rationale: _TBD_ +- Next cohort/config delta: _TBD_ + +## Diagram/Protocol Impact Review + +- Reviewed: `docs/architecture/AGENT_DIAGRAM.md`, `docs/architecture/GATEWAY_SESSIONS_AND_QUEUE.md`, `docs/api/PROTOCOL.md` +- Result: no runtime message-flow or protocol-shape changes; no Mermaid topology update required for this evaluation-tooling phase. diff --git a/docs/plans/state.json b/docs/plans/state.json index 1b97795..e6e2eff 100644 --- a/docs/plans/state.json +++ b/docs/plans/state.json @@ -3,6 +3,26 @@ "updated_at": "2026-02-24", "description": "Tracks the status of all Flynn plans and implementation phases", "plans": { + "pi-embedded-backend-canary-evaluation-phase": { + "status": "in_progress", + "date": "2026-02-24", + "updated": "2026-02-24", + "summary": "Started formal Pi embedded canary evaluation with reproducible audit-log summarization (route/success/fallback/session latency), explicit rollout gate thresholds, and a decision record template covering reliability, latency, fallback taxonomy, and tool-compat findings.", + "files_modified": [ + "src/audit/backendCanarySummary.ts", + "src/audit/backendCanarySummary.test.ts", + "scripts/summarize-backend-canary.ts", + "package.json", + "README.md", + "docs/plans/pi_embedded_evaluation.md", + "docs/plans/artifacts/.gitkeep", + "docs/architecture/AGENT_DIAGRAM.md", + "docs/architecture/GATEWAY_SESSIONS_AND_QUEUE.md", + "docs/api/PROTOCOL.md", + "docs/plans/state.json" + ], + "test_status": "pnpm test:run src/audit/backendCanarySummary.test.ts + pnpm typecheck + pnpm audit:backend-canary (smoke run) passing; pnpm lint unchanged warnings-only baseline" + }, "pi-embedded-backend-canary-spike": { "status": "completed", "date": "2026-02-24", @@ -6430,7 +6450,7 @@ } }, "overall_progress": { - "total_test_count": 1989, + "total_test_count": 1992, "all_tests_passing": true, "p0_completion": "3/3 (100%)", "p1_completion": "4/4 (100%)", @@ -6461,7 +6481,8 @@ "native_audio_support": "completed — smart routing for native audio (Gemini/OpenAI/GitHub) vs Whisper transcription fallback, plus 2026-02-23 arg hydration hardening, tool.args_rewritten audit metric, transient fetch retry/timeout hardening, localhost->127.0.0.1 fallback for transcription endpoint connectivity, and whisper docker-compose entrypoint arg fix for port 18801", "remaining_phases_completion": "Phase 1: 3/3 (100%) — context levels, command registry, memory structure. Phase 2: 3/3 (100%) — component registry, confidence routing, history index. Phase 3: 2/2 (100%) — adaptive memory/compaction, truthfulness/autonomy hardening", "next_up": "Track OpenClaw evolution regularly for inspiration and feature ideas", - "pi_embedded_canary_spike": "completed — added optional pi_embedded backend adapter, canary-safe no-tools routing guard, backend success/fallback latency audit events, and docs/diagram updates while native remains default" + "pi_embedded_canary_spike": "completed — added optional pi_embedded backend adapter, canary-safe no-tools routing guard, backend success/fallback latency audit events, and docs/diagram updates while native remains default", + "pi_embedded_evaluation_phase": "in progress — added audit-backed canary summarizer + gate thresholds + decision template for reliability/latency/tool-compat rollout decisions" }, "soul_md_and_cron_create": { "date": "2026-02-11",