William Valentin
115 lines
3.1 KiB
Markdown
115 lines
3.1 KiB
Markdown
# Pi Embedded Canary Evaluation (Phase 2)
|
|
|
|
Status: in progress
|
|
Owner: Flynn maintainers
|
|
Started: 2026-02-24
|
|
|
|
## Goal
|
|
|
|
Close the canary spike with a formal, repeatable evaluation and an explicit rollout decision (`expand`, `hold`, or `rollback`).
|
|
|
|
## Scope
|
|
|
|
- Target backend: `pi_embedded`
|
|
- Target cohort (current): `telegram:8367012007`
|
|
- Baseline backend: `native`
|
|
- Data source: audit events (`backend.route`, `backend.success`, `backend.fallback`, `session.message`)
|
|
|
|
## Pass/Fail Gate
|
|
|
|
Use the same thresholds for every evaluation window.
|
|
|
|
| Metric | Gate |
|
|
| --- | --- |
|
|
| Completion rate delta (target - baseline) | >= -2.00pp |
|
|
| P50 latency delta (target - baseline) | <= +250ms |
|
|
| P95 latency delta (target - baseline) | <= +700ms |
|
|
| External fallback rate (`pi_embedded`) | <= 5.00% |
|
|
| Guardrail escapes | 0 unresolved |
|
|
|
|
Notes:
|
|
- Completion rate and latency are computed from route-to-assistant turn timings.
|
|
- Fallback rate is computed from `backend.success` + `backend.fallback` attempt outcomes.
|
|
- Guardrail escapes are reviewed from `backend.route.source == forced_native_guard` + operator incident review.
|
|
|
|
## How To Run
|
|
|
|
Run a canary summary for the current cohort:
|
|
|
|
```bash
|
|
pnpm audit:backend-canary \
|
|
--audit ~/.local/share/flynn/audit.log \
|
|
--backend pi_embedded \
|
|
--baseline native \
|
|
--session telegram:8367012007 \
|
|
--format markdown
|
|
```
|
|
|
|
Run with gate evaluation and emit JSON artifact:
|
|
|
|
```bash
|
|
pnpm audit:backend-canary \
|
|
--audit ~/.local/share/flynn/audit.log \
|
|
--backend pi_embedded \
|
|
--baseline native \
|
|
--session telegram:8367012007 \
|
|
--format json \
|
|
--out docs/plans/artifacts/pi_embedded_eval_latest.json \
|
|
--gate-max-completion-drop-pp 2 \
|
|
--gate-max-p50-latency-increase-ms 250 \
|
|
--gate-max-p95-latency-increase-ms 700 \
|
|
--gate-max-fallback-rate-pct 5
|
|
```
|
|
|
|
## Evaluation Log
|
|
|
|
### Window A
|
|
|
|
- Dates: _TBD_
|
|
- Route volume: _TBD_
|
|
- Summary artifact: _TBD_
|
|
|
|
| Check | Result | Notes |
|
|
| --- | --- | --- |
|
|
| Completion rate delta | _TBD_ | |
|
|
| P50 latency delta | _TBD_ | |
|
|
| P95 latency delta | _TBD_ | |
|
|
| Fallback rate | _TBD_ | |
|
|
| Guardrail escapes | _TBD_ | |
|
|
|
|
### Window B
|
|
|
|
- Dates: _TBD_
|
|
- Route volume: _TBD_
|
|
- Summary artifact: _TBD_
|
|
|
|
| Check | Result | Notes |
|
|
| --- | --- | --- |
|
|
| Completion rate delta | _TBD_ | |
|
|
| P50 latency delta | _TBD_ | |
|
|
| P95 latency delta | _TBD_ | |
|
|
| Fallback rate | _TBD_ | |
|
|
| Guardrail escapes | _TBD_ | |
|
|
|
|
## Tool Compatibility Findings
|
|
|
|
Track all tool-adjacent/risky prompts that were force-routed to native (`no_tools_mode`) and any misses.
|
|
|
|
| Class | Observed behavior | Action |
|
|
| --- | --- | --- |
|
|
| Tool-adjacent prompts | _TBD_ | |
|
|
| Capability-query prompts | _TBD_ | |
|
|
| Attachments-present turns | _TBD_ | |
|
|
|
|
## Decision Record
|
|
|
|
- Decision date: _TBD_
|
|
- Decision: _expand | hold | rollback_
|
|
- Rationale: _TBD_
|
|
- Next cohort/config delta: _TBD_
|
|
|
|
## Diagram/Protocol Impact Review
|
|
|
|
- Reviewed: `docs/architecture/AGENT_DIAGRAM.md`, `docs/architecture/GATEWAY_SESSIONS_AND_QUEUE.md`, `docs/api/PROTOCOL.md`
|
|
- Result: no runtime message-flow or protocol-shape changes; no Mermaid topology update required for this evaluation-tooling phase.
|