will/flynn
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(eval): enforce min-sample gates in canary artifacts and decision log

Browse Source
This commit is contained in:
William Valentin
2026-02-23 22:34:54 -08:00
parent 34d1562ce8
commit 2d42f65b9f
7 changed files with 76 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/pi_embedded_evaluation.md
+22 -7
View File
@@ -21,6 +21,9 @@ Use the same thresholds for every evaluation window.
| Metric | Gate |
| --- | --- |
| Minimum target routes | >= 8 |
| Minimum baseline routes | >= 2 |
| Minimum target external attempts | >= 8 |
| Completion rate delta (target - baseline) | >= -2.00pp |
| P50 latency delta (target - baseline) | <= +250ms |
| P95 latency delta (target - baseline) | <= +700ms |
@@ -42,6 +45,9 @@ pnpm audit:backend-canary \
--backend pi_embedded \
--baseline native \
--session telegram:8367012007 \
--gate-min-target-routes 8 \
--gate-min-baseline-routes 2 \
--gate-min-target-attempts 8 \
--format markdown
```
@@ -55,6 +61,9 @@ pnpm audit:backend-canary \
--session telegram:8367012007 \
--format json \
--out docs/plans/artifacts/pi_embedded_eval_latest.json \
--gate-min-target-routes 8 \
--gate-min-baseline-routes 2 \
--gate-min-target-attempts 8 \
--gate-max-completion-drop-pp 2 \
--gate-max-p50-latency-increase-ms 250 \
--gate-max-p95-latency-increase-ms 700 \
@@ -73,6 +82,9 @@ pnpm audit:backend-canary \
| Check | Result | Notes |
| --- | --- | --- |
| Minimum target routes | 8 (pass) | gate >= 8 |
| Minimum baseline routes | 2 (pass) | gate >= 2 |
| Minimum target external attempts | 8 (pass) | gate >= 8 |
| Completion rate delta | 0.00pp (pass) | target 100.00% vs baseline 100.00% |
| P50 latency delta | +259ms (fail) | gate <= +250ms |
| P95 latency delta | +5695ms (fail) | gate <= +700ms |
@@ -89,6 +101,9 @@ pnpm audit:backend-canary \
| Check | Result | Notes |
| --- | --- | --- |
| Minimum target routes | 6 (fail) | gate >= 8 |
| Minimum baseline routes | 0 (fail) | gate >= 2 |
| Minimum target external attempts | 6 (fail) | gate >= 8 |
| Completion rate delta | n/a (insufficient baseline) | no native-routed turns in this slice |
| P50 latency delta | n/a (insufficient baseline) | no native-routed turns in this slice |
| P95 latency delta | n/a (insufficient baseline) | no native-routed turns in this slice |
@@ -101,19 +116,19 @@ Track all tool-adjacent/risky prompts that were force-routed to native (`no_tool
| Class | Observed behavior | Action |
| --- | --- | --- |
| Tool-adjacent prompts | Not observed in Window A (`forced_native_guard` count 0). | Collect dedicated tool-adjacent prompts in Window B to validate `no_tools_mode` behavior. |
| Capability-query prompts | Not observed in Window A (`guard_reason=capability_query` count 0). | Add explicit capability-query probes in Window B. |
| Attachments-present turns | Not observed in Window A (`guard_reason=attachments_present` count 0). | Add attachment turns in Window B. |
| Tool-adjacent prompts | Not observed in Window A/B (`forced_native_guard` count 0). | Run controlled probe turns that should trigger `pi_no_tools_mode`. |
| Capability-query prompts | Not observed in Window A/B (`guard_reason=capability_query` count 0). | Run explicit capability-query probe prompts and confirm forced-native routing. |
| Attachments-present turns | Not observed in Window A/B (`guard_reason=attachments_present` count 0). | Run attachment-bearing probe turns and confirm forced-native routing. |
## Decision Record
- Decision date: February 24, 2026
- Decision: `hold` (no cohort expansion yet)
- Rationale: Window A fails 3/4 numeric gates (p50 delta, p95 delta, fallback rate) with only 10 total routed turns, including two concrete fallback failure modes:
- module session factory mismatch
- no assistant text returned from Pi runtime
Window B shows fallback recovery (0%) in a post-fallback slice but cannot evaluate delta gates because it contains no baseline native routes.
- Next cohort/config delta: none until an additional baseline-balanced window confirms delta gates and guardrail coverage probes are completed.
- `pi_module_interface`
- `empty_assistant_text`
Window B shows fallback recovery (0%) in a post-fallback slice but fails minimum sample thresholds and has no native baseline routes for delta-gate evaluation.
- Next cohort/config delta: none until a baseline-balanced window meets minimum sample thresholds and guardrail coverage probes are completed.
## Diagram/Protocol Impact Review