docs(plan): add pi canary evaluation gate and tracking (diagram review no topology change)
This commit is contained in:
William Valentin
@@ -0,0 +1,114 @@
|
||||
# Pi Embedded Canary Evaluation (Phase 2)
|
||||
|
||||
Status: in progress
|
||||
Owner: Flynn maintainers
|
||||
Started: 2026-02-24
|
||||
|
||||
## Goal
|
||||
|
||||
Close the canary spike with a formal, repeatable evaluation and an explicit rollout decision (`expand`, `hold`, or `rollback`).
|
||||
|
||||
## Scope
|
||||
|
||||
- Target backend: `pi_embedded`
|
||||
- Target cohort (current): `telegram:8367012007`
|
||||
- Baseline backend: `native`
|
||||
- Data source: audit events (`backend.route`, `backend.success`, `backend.fallback`, `session.message`)
|
||||
|
||||
## Pass/Fail Gate
|
||||
|
||||
Use the same thresholds for every evaluation window.
|
||||
|
||||
| Metric | Gate |
|
||||
| --- | --- |
|
||||
| Completion rate delta (target - baseline) | >= -2.00pp |
|
||||
| P50 latency delta (target - baseline) | <= +250ms |
|
||||
| P95 latency delta (target - baseline) | <= +700ms |
|
||||
| External fallback rate (`pi_embedded`) | <= 5.00% |
|
||||
| Guardrail escapes | 0 unresolved |
|
||||
|
||||
Notes:
|
||||
- Completion rate and latency are computed from route-to-assistant turn timings.
|
||||
- Fallback rate is computed from `backend.success` + `backend.fallback` attempt outcomes.
|
||||
- Guardrail escapes are reviewed from `backend.route.source == forced_native_guard` + operator incident review.
|
||||
|
||||
## How To Run
|
||||
|
||||
Run a canary summary for the current cohort:
|
||||
|
||||
```bash
|
||||
pnpm audit:backend-canary \
|
||||
--audit ~/.local/share/flynn/audit.log \
|
||||
--backend pi_embedded \
|
||||
--baseline native \
|
||||
--session telegram:8367012007 \
|
||||
--format markdown
|
||||
```
|
||||
|
||||
Run with gate evaluation and emit JSON artifact:
|
||||
|
||||
```bash
|
||||
pnpm audit:backend-canary \
|
||||
--audit ~/.local/share/flynn/audit.log \
|
||||
--backend pi_embedded \
|
||||
--baseline native \
|
||||
--session telegram:8367012007 \
|
||||
--format json \
|
||||
--out docs/plans/artifacts/pi_embedded_eval_latest.json \
|
||||
--gate-max-completion-drop-pp 2 \
|
||||
--gate-max-p50-latency-increase-ms 250 \
|
||||
--gate-max-p95-latency-increase-ms 700 \
|
||||
--gate-max-fallback-rate-pct 5
|
||||
```
|
||||
|
||||
## Evaluation Log
|
||||
|
||||
### Window A
|
||||
|
||||
- Dates: _TBD_
|
||||
- Route volume: _TBD_
|
||||
- Summary artifact: _TBD_
|
||||
|
||||
| Check | Result | Notes |
|
||||
| --- | --- | --- |
|
||||
| Completion rate delta | _TBD_ | |
|
||||
| P50 latency delta | _TBD_ | |
|
||||
| P95 latency delta | _TBD_ | |
|
||||
| Fallback rate | _TBD_ | |
|
||||
| Guardrail escapes | _TBD_ | |
|
||||
|
||||
### Window B
|
||||
|
||||
- Dates: _TBD_
|
||||
- Route volume: _TBD_
|
||||
- Summary artifact: _TBD_
|
||||
|
||||
| Check | Result | Notes |
|
||||
| --- | --- | --- |
|
||||
| Completion rate delta | _TBD_ | |
|
||||
| P50 latency delta | _TBD_ | |
|
||||
| P95 latency delta | _TBD_ | |
|
||||
| Fallback rate | _TBD_ | |
|
||||
| Guardrail escapes | _TBD_ | |
|
||||
|
||||
## Tool Compatibility Findings
|
||||
|
||||
Track all tool-adjacent/risky prompts that were force-routed to native (`no_tools_mode`) and any misses.
|
||||
|
||||
| Class | Observed behavior | Action |
|
||||
| --- | --- | --- |
|
||||
| Tool-adjacent prompts | _TBD_ | |
|
||||
| Capability-query prompts | _TBD_ | |
|
||||
| Attachments-present turns | _TBD_ | |
|
||||
|
||||
## Decision Record
|
||||
|
||||
- Decision date: _TBD_
|
||||
- Decision: _expand | hold | rollback_
|
||||
- Rationale: _TBD_
|
||||
- Next cohort/config delta: _TBD_
|
||||
|
||||
## Diagram/Protocol Impact Review
|
||||
|
||||
- Reviewed: `docs/architecture/AGENT_DIAGRAM.md`, `docs/architecture/GATEWAY_SESSIONS_AND_QUEUE.md`, `docs/api/PROTOCOL.md`
|
||||
- Result: no runtime message-flow or protocol-shape changes; no Mermaid topology update required for this evaluation-tooling phase.
|
||||
+23
-2
@@ -3,6 +3,26 @@
|
||||
"updated_at": "2026-02-24",
|
||||
"description": "Tracks the status of all Flynn plans and implementation phases",
|
||||
"plans": {
|
||||
"pi-embedded-backend-canary-evaluation-phase": {
|
||||
"status": "in_progress",
|
||||
"date": "2026-02-24",
|
||||
"updated": "2026-02-24",
|
||||
"summary": "Started formal Pi embedded canary evaluation with reproducible audit-log summarization (route/success/fallback/session latency), explicit rollout gate thresholds, and a decision record template covering reliability, latency, fallback taxonomy, and tool-compat findings.",
|
||||
"files_modified": [
|
||||
"src/audit/backendCanarySummary.ts",
|
||||
"src/audit/backendCanarySummary.test.ts",
|
||||
"scripts/summarize-backend-canary.ts",
|
||||
"package.json",
|
||||
"README.md",
|
||||
"docs/plans/pi_embedded_evaluation.md",
|
||||
"docs/plans/artifacts/.gitkeep",
|
||||
"docs/architecture/AGENT_DIAGRAM.md",
|
||||
"docs/architecture/GATEWAY_SESSIONS_AND_QUEUE.md",
|
||||
"docs/api/PROTOCOL.md",
|
||||
"docs/plans/state.json"
|
||||
],
|
||||
"test_status": "pnpm test:run src/audit/backendCanarySummary.test.ts + pnpm typecheck + pnpm audit:backend-canary (smoke run) passing; pnpm lint unchanged warnings-only baseline"
|
||||
},
|
||||
"pi-embedded-backend-canary-spike": {
|
||||
"status": "completed",
|
||||
"date": "2026-02-24",
|
||||
@@ -6430,7 +6450,7 @@
|
||||
}
|
||||
},
|
||||
"overall_progress": {
|
||||
"total_test_count": 1989,
|
||||
"total_test_count": 1992,
|
||||
"all_tests_passing": true,
|
||||
"p0_completion": "3/3 (100%)",
|
||||
"p1_completion": "4/4 (100%)",
|
||||
@@ -6461,7 +6481,8 @@
|
||||
"native_audio_support": "completed — smart routing for native audio (Gemini/OpenAI/GitHub) vs Whisper transcription fallback, plus 2026-02-23 arg hydration hardening, tool.args_rewritten audit metric, transient fetch retry/timeout hardening, localhost->127.0.0.1 fallback for transcription endpoint connectivity, and whisper docker-compose entrypoint arg fix for port 18801",
|
||||
"remaining_phases_completion": "Phase 1: 3/3 (100%) — context levels, command registry, memory structure. Phase 2: 3/3 (100%) — component registry, confidence routing, history index. Phase 3: 2/2 (100%) — adaptive memory/compaction, truthfulness/autonomy hardening",
|
||||
"next_up": "Track OpenClaw evolution regularly for inspiration and feature ideas",
|
||||
"pi_embedded_canary_spike": "completed — added optional pi_embedded backend adapter, canary-safe no-tools routing guard, backend success/fallback latency audit events, and docs/diagram updates while native remains default"
|
||||
"pi_embedded_canary_spike": "completed — added optional pi_embedded backend adapter, canary-safe no-tools routing guard, backend success/fallback latency audit events, and docs/diagram updates while native remains default",
|
||||
"pi_embedded_evaluation_phase": "in progress — added audit-backed canary summarizer + gate thresholds + decision template for reliability/latency/tool-compat rollout decisions"
|
||||
},
|
||||
"soul_md_and_cron_create": {
|
||||
"date": "2026-02-11",
|
||||
|
||||
Reference in New Issue
Block a user