will/flynn
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(eval): record guard-coverage preprobe window and updated gates

Browse Source
This commit is contained in:
William Valentin
2026-02-23 22:38:18 -08:00
parent c5b310c852
commit 7b80c1e7a4
9 changed files with 291 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/pi_embedded_evaluation.md
+45
View File
@@ -24,6 +24,7 @@ Use the same thresholds for every evaluation window.
| Minimum target routes | >= 8 |
| Minimum baseline routes | >= 2 |
| Minimum target external attempts | >= 8 |
| Minimum guard coverage (probe window) | `pi_no_tools_mode >= 1`, `capability_query >= 1`, `attachments_present >= 1` |
| Completion rate delta (target - baseline) | >= -2.00pp |
| P50 latency delta (target - baseline) | <= +250ms |
| P95 latency delta (target - baseline) | <= +700ms |
@@ -34,6 +35,7 @@ Notes:
- Completion rate and latency are computed from route-to-assistant turn timings.
- Fallback rate is computed from `backend.success` + `backend.fallback` attempt outcomes.
- Guardrail escapes are reviewed from `backend.route.source == forced_native_guard` + operator incident review.
- Guard-coverage minimums are enforced for controlled probe windows, not passive traffic slices.
## How To Run
@@ -70,6 +72,27 @@ pnpm audit:backend-canary \
--gate-max-fallback-rate-pct 5
```
Run controlled probe-window evaluation (guard coverage required):
```bash
pnpm audit:backend-canary \
--audit ~/.local/share/flynn/audit.log \
--backend pi_embedded \
--baseline native \
--session telegram:8367012007 \
--gate-min-target-routes 8 \
--gate-min-baseline-routes 2 \
--gate-min-target-attempts 8 \
--gate-min-guard-pi-no-tools-count 1 \
--gate-min-guard-capability-query-count 1 \
--gate-min-guard-attachments-present-count 1 \
--gate-max-completion-drop-pp 2 \
--gate-max-p50-latency-increase-ms 250 \
--gate-max-p95-latency-increase-ms 700 \
--gate-max-fallback-rate-pct 5 \
--format markdown
```
## Evaluation Log
### Window A
@@ -110,6 +133,27 @@ pnpm audit:backend-canary \
| Fallback rate | 0.00% (pass) | 0 fallbacks / 6 attempts |
| Guardrail escapes | none observed (provisional pass) | no `forced_native_guard` events in this window |
### Window C (Guard Coverage Pre-Probe Baseline)
- Dates: February 24, 2026 (same full Window A slice; guard-coverage gates enabled)
- Route volume: 10 total routes (`pi_embedded`: 8, `native`: 2)
- Summary artifacts:
- `docs/plans/artifacts/pi_embedded_eval_window_c_2026-02-24_guard_preprobe.md`
- `docs/plans/artifacts/pi_embedded_eval_window_c_2026-02-24_guard_preprobe.json`
| Check | Result | Notes |
| --- | --- | --- |
| Minimum target routes | 8 (pass) | gate >= 8 |
| Minimum baseline routes | 2 (pass) | gate >= 2 |
| Minimum target external attempts | 8 (pass) | gate >= 8 |
| Minimum `pi_no_tools_mode` guard hits | 0 (fail) | gate >= 1 |
| Minimum `capability_query` guard hits | 0 (fail) | gate >= 1 |
| Minimum `attachments_present` guard hits | 0 (fail) | gate >= 1 |
| Completion rate delta | 0.00pp (pass) | target 100.00% vs baseline 100.00% |
| P50 latency delta | +259ms (fail) | gate <= +250ms |
| P95 latency delta | +5695ms (fail) | gate <= +700ms |
| Fallback rate | 25.00% (fail) | 2 fallbacks / 8 attempts; gate <= 5.00% |
## Tool Compatibility Findings
Track all tool-adjacent/risky prompts that were force-routed to native (`no_tools_mode`) and any misses.
@@ -125,6 +169,7 @@ Track all tool-adjacent/risky prompts that were force-routed to native (`no_tool
- Decision date: February 24, 2026
- Decision: `hold` (no cohort expansion yet)
- Rationale: Window A fails 3/4 numeric gates (p50 delta, p95 delta, fallback rate) with only 10 total routed turns, including two concrete fallback failure modes:
and Window C pre-probe baseline confirms missing guard-coverage evidence (`pi_no_tools_mode`, `capability_query`, `attachments_present` all at 0).
- `pi_module_interface`
- `empty_assistant_text`
Window B shows fallback recovery (0%) in a post-fallback slice but fails minimum sample thresholds and has no native baseline routes for delta-gate evaluation.