Pi Embedded Canary Evaluation (Phase 2)

Status: in progress
Owner: Flynn maintainers
Started: 2026-02-24

Goal

Close the canary spike with a formal, repeatable evaluation and an explicit rollout decision (expand, hold, or rollback).

Scope

Target backend: pi_embedded
Target cohort (current): telegram:8367012007
Baseline backend: native
Data source: audit events (backend.route, backend.success, backend.fallback, session.message)

Pass/Fail Gate

Use the same thresholds for every evaluation window.

Metric	Gate
Completion rate delta (target - baseline)	>= -2.00pp
P50 latency delta (target - baseline)	<= +250ms
P95 latency delta (target - baseline)	<= +700ms
External fallback rate (`pi_embedded`)	<= 5.00%
Guardrail escapes	0 unresolved

Notes:

Completion rate and latency are computed from route-to-assistant turn timings.
Fallback rate is computed from backend.success + backend.fallback attempt outcomes.
Guardrail escapes are reviewed from backend.route.source == forced_native_guard + operator incident review.

How To Run

Run a canary summary for the current cohort:

pnpm audit:backend-canary \
  --audit ~/.local/share/flynn/audit.log \
  --backend pi_embedded \
  --baseline native \
  --session telegram:8367012007 \
  --format markdown

Run with gate evaluation and emit JSON artifact:

pnpm audit:backend-canary \
  --audit ~/.local/share/flynn/audit.log \
  --backend pi_embedded \
  --baseline native \
  --session telegram:8367012007 \
  --format json \
  --out docs/plans/artifacts/pi_embedded_eval_latest.json \
  --gate-max-completion-drop-pp 2 \
  --gate-max-p50-latency-increase-ms 250 \
  --gate-max-p95-latency-increase-ms 700 \
  --gate-max-fallback-rate-pct 5

Evaluation Log

Window A

Dates: February 24, 2026 (05:29:49Z to 06:26:20Z)
Route volume: 10 total routes (pi_embedded: 8, native: 2)
Summary artifacts:
- docs/plans/artifacts/pi_embedded_eval_window_a_2026-02-24.md
- docs/plans/artifacts/pi_embedded_eval_window_a_2026-02-24.json

Check	Result	Notes
Completion rate delta	0.00pp (pass)	target 100.00% vs baseline 100.00%
P50 latency delta	+259ms (fail)	gate <= +250ms
P95 latency delta	+5695ms (fail)	gate <= +700ms
Fallback rate	25.00% (fail)	2 fallbacks / 8 attempts; gate <= 5.00%
Guardrail escapes	none observed (provisional pass)	no `forced_native_guard` events in this window

Window B

Dates: February 24, 2026 (since 06:14:00Z; post-initial-fallback slice)
Route volume: 6 total routes (pi_embedded: 6, native: 0)
Summary artifacts:
- docs/plans/artifacts/pi_embedded_eval_window_b_2026-02-24_post_fallbacks.md
- docs/plans/artifacts/pi_embedded_eval_window_b_2026-02-24_post_fallbacks.json

Check	Result	Notes
Completion rate delta	n/a (insufficient baseline)	no native-routed turns in this slice
P50 latency delta	n/a (insufficient baseline)	no native-routed turns in this slice
P95 latency delta	n/a (insufficient baseline)	no native-routed turns in this slice
Fallback rate	0.00% (pass)	0 fallbacks / 6 attempts
Guardrail escapes	none observed (provisional pass)	no `forced_native_guard` events in this window

Tool Compatibility Findings

Track all tool-adjacent/risky prompts that were force-routed to native (no_tools_mode) and any misses.

Class	Observed behavior	Action
Tool-adjacent prompts	Not observed in Window A (`forced_native_guard` count 0).	Collect dedicated tool-adjacent prompts in Window B to validate `no_tools_mode` behavior.
Capability-query prompts	Not observed in Window A (`guard_reason=capability_query` count 0).	Add explicit capability-query probes in Window B.
Attachments-present turns	Not observed in Window A (`guard_reason=attachments_present` count 0).	Add attachment turns in Window B.

Decision Record

Decision date: February 24, 2026
Decision: hold (no cohort expansion yet)
Rationale: Window A fails 3/4 numeric gates (p50 delta, p95 delta, fallback rate) with only 10 total routed turns, including two concrete fallback failure modes:
- module session factory mismatch
- no assistant text returned from Pi runtime Window B shows fallback recovery (0%) in a post-fallback slice but cannot evaluate delta gates because it contains no baseline native routes.
Next cohort/config delta: none until an additional baseline-balanced window confirms delta gates and guardrail coverage probes are completed.

Diagram/Protocol Impact Review

Reviewed: docs/architecture/AGENT_DIAGRAM.md, docs/architecture/GATEWAY_SESSIONS_AND_QUEUE.md, docs/api/PROTOCOL.md
Result: no runtime message-flow or protocol-shape changes; no Mermaid topology update required for this evaluation-tooling phase.

4.9 KiB Raw Blame History