will/flynn
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(observability): capture gateway-origin phase0 live baseline window

Browse Source
This commit is contained in:
William Valentin
2026-02-26 23:50:28 -08:00
parent 4b07a1f166
commit 5a34e986bf
10 changed files with 200 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/2026-02-25-phase0-instrumentation-ticket-checklist.md
+1 -1
View File
@@ -203,7 +203,7 @@ Phase 0 is complete when:
2. A baseline summary artifact is generated and committed under `docs/plans/artifacts/`.
3. No user-visible response behavior changed compared to pre-phase baseline.
Follow-up status (2026-02-27): live channel-session artifacts now exist under `docs/plans/artifacts/phase0_baseline_live_2026-02-27.*` via `pnpm audit:phase0-baseline:live` (anonymized IDs); gateway-origin live samples remain a future slice.
Follow-up status (2026-02-27): live channel-session artifacts exist under `docs/plans/artifacts/phase0_baseline_live_2026-02-27.*` via `pnpm audit:phase0-baseline:live` (anonymized IDs), and a second gateway-origin live window (including `run.cancel` + `cancel_requested`/`cancelled`) exists under `docs/plans/artifacts/phase0_baseline_live_gateway_2026-02-27.*`.
## Subagent Model Assignment Plan
docs/plans/artifacts/phase0_baseline_live_2026-02-27.json
+2 -2
View File
@@ -1,7 +1,7 @@
{
"generated_at": "2026-02-27T07:39:16.384Z",
"generated_at": "2026-02-27T07:49:58.821Z",
"source_audit_path": "~/.local/share/flynn/audit.log",
"source_event_count": 88,
"source_event_count": 94,
"sampled_event_count": 88,
"filters": {
"sources": [
docs/plans/artifacts/phase0_baseline_live_gateway_2026-02-27.json
+94
View File
@@ -0,0 +1,94 @@
{
"generated_at": "2026-02-27T07:47:41.346Z",
"source_audit_path": "~/.local/share/flynn/audit.log",
"source_event_count": 6,
"sampled_event_count": 6,
"filters": {
"since_ms": 1772178440693,
"until_ms": 1772178442694,
"sources": [
"gateway"
],
"exclude_session_substrings": [
"probe"
],
"anonymized_identifiers": true
},
"options": {
"sources": [
"gateway"
],
"maxSessions": 20,
"maxChannels": 20,
"maxSkipReasons": 10
},
"summary": {
"event_counts": {
"run_state": 5,
"run_cancel": 1,
"reaction_match": 0,
"reaction_skip": 0
},
"run_outcomes": {
"overall": {
"total_outcomes": 2,
"complete": 1,
"cancelled": 1,
"error": 0,
"cancel_requested": 1,
"start": 2,
"completion_rate_pct": 50,
"cancel_rate_pct": 50,
"error_rate_pct": 0
},
"by_channel": [
{
"key": "ws",
"stats": {
"total_outcomes": 2,
"complete": 1,
"cancelled": 1,
"error": 0,
"cancel_requested": 1,
"start": 2,
"completion_rate_pct": 50,
"cancel_rate_pct": 50,
"error_rate_pct": 0
}
}
],
"by_session": [
{
"key": "session_67024a716ed2",
"stats": {
"total_outcomes": 2,
"complete": 1,
"cancelled": 1,
"error": 0,
"cancel_requested": 1,
"start": 2,
"completion_rate_pct": 50,
"cancel_rate_pct": 50,
"error_rate_pct": 0
}
}
]
},
"cancel_latency_ms": {
"count": 1,
"avg_ms": 0,
"p50_ms": 0,
"p95_ms": 0,
"min_ms": 0,
"max_ms": 0
},
"reactions": {
"matched": 0,
"skipped": 0,
"total": 0,
"match_rate_pct": null,
"skip_rate_pct": null,
"skip_reasons": []
}
}
}
docs/plans/artifacts/phase0_baseline_live_gateway_2026-02-27.jsonl
+6
View File
@@ -0,0 +1,6 @@
{"level":"info","event_type":"run.state","event":{"session_id":"session_67024a716ed2","channel":"ws","sender":"sender_776ae1d44923","source":"gateway","state":"start","request_id":"request_e7102f267cb2"},"timestamp":1772178441693}
{"level":"info","event_type":"run.cancel","event":{"session_id":"session_67024a716ed2","channel":"ws","sender":"sender_776ae1d44923","source":"gateway","requested":true,"acknowledged":true,"request_id":"request_61e1e51ee26b","latency_ms":0},"timestamp":1772178441693}
{"level":"info","event_type":"run.state","event":{"session_id":"session_67024a716ed2","channel":"ws","sender":"sender_776ae1d44923","source":"gateway","state":"cancel_requested","request_id":"request_61e1e51ee26b","duration_ms":0},"timestamp":1772178441694}
{"level":"info","event_type":"run.state","event":{"session_id":"session_67024a716ed2","channel":"ws","sender":"sender_776ae1d44923","source":"gateway","state":"cancelled","request_id":"request_e7102f267cb2","duration_ms":1},"timestamp":1772178441694}
{"level":"info","event_type":"run.state","event":{"session_id":"session_67024a716ed2","channel":"ws","sender":"sender_776ae1d44923","source":"gateway","state":"start","request_id":"request_897903dab9c4"},"timestamp":1772178441694}
{"level":"info","event_type":"run.state","event":{"session_id":"session_67024a716ed2","channel":"ws","sender":"sender_776ae1d44923","source":"gateway","state":"complete","request_id":"request_897903dab9c4","duration_ms":0},"timestamp":1772178441694}
docs/plans/artifacts/phase0_baseline_live_gateway_2026-02-27.md
+50
View File
@@ -0,0 +1,50 @@
# Phase 0 Baseline Telemetry Summary
- Run state events: 5
- Run cancel events: 1
- Reaction matches: 0
- Reaction skips: 0
- Sources: gateway
## Run Outcomes (Overall)
- Total outcomes: 2
- Complete: 1 (50.00%)
- Cancelled: 1 (50.00%)
- Errors: 0 (0.00%)
- Cancel requested: 1
- Starts: 2
## Run Outcomes by Channel
| Channel | Outcomes | Complete | Cancelled | Error | Complete % | Cancel % | Error % | Cancel Req | Starts |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| ws | 2 | 1 | 1 | 0 | 50.00% | 50.00% | 0.00% | 1 | 2 |
## Run Outcomes by Session
| Session | Outcomes | Complete | Cancelled | Error | Complete % | Cancel % | Error % | Cancel Req | Starts |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| session_67024a716ed2 | 2 | 1 | 1 | 0 | 50.00% | 50.00% | 0.00% | 1 | 2 |
## Cancel Latency
- Count: 1
- Avg: 0ms
- P50: 0ms
- P95: 0ms
- Min: 0ms
- Max: 0ms
## Reaction Decisions
- Matched: 0 (n/a)
- Skipped: 0 (n/a)
### Skip Reasons
| Reason | Count | Percent |
| --- | ---: | ---: |
| _none_ | 0 | 0.00% |
docs/plans/state.json
+30 -6
View File
@@ -67,7 +67,7 @@
"status": "completed",
"date": "2026-02-25",
"updated": "2026-02-27",
"summary": "Updated protocol/docs/diagrams for phase-0 telemetry fields, documented baseline workflow, and replaced the original probe-only baseline workflow with anonymized live channel-session audit artifacts (`phase0_baseline_live_2026-02-27.*`).",
"summary": "Updated protocol/docs/diagrams for phase-0 telemetry fields, documented baseline workflow, replaced the original probe-only baseline workflow with anonymized live channel-session audit artifacts (`phase0_baseline_live_2026-02-27.*`), and added a second gateway-origin live window artifact set (`phase0_baseline_live_gateway_2026-02-27.*`) including cancel-path telemetry.",
"files_modified": [
"README.md",
"docs/api/PROTOCOL.md",
@@ -77,15 +77,18 @@
"docs/plans/artifacts/phase0_baseline_live_2026-02-27.jsonl",
"docs/plans/artifacts/phase0_baseline_live_2026-02-27.md",
"docs/plans/artifacts/phase0_baseline_live_2026-02-27.json",
"docs/plans/artifacts/phase0_baseline_live_gateway_2026-02-27.jsonl",
"docs/plans/artifacts/phase0_baseline_live_gateway_2026-02-27.md",
"docs/plans/artifacts/phase0_baseline_live_gateway_2026-02-27.json",
"docs/plans/state.json"
],
"test_status": "pnpm audit:phase0-baseline:live + pnpm test:run src/audit/phase0LiveBaseline.test.ts src/audit/phase0BaselineSummary.test.ts + pnpm typecheck passing"
"test_status": "pnpm audit:phase0-baseline:live + node --import tsx/esm scripts/capture-phase0-live-baseline.ts --audit ~/.local/share/flynn/audit.log --source gateway --since 1772178440693 --until 1772178442694 --sample-out docs/plans/artifacts/phase0_baseline_live_gateway_2026-02-27.jsonl --summary-json-out docs/plans/artifacts/phase0_baseline_live_gateway_2026-02-27.json --summary-md-out docs/plans/artifacts/phase0_baseline_live_gateway_2026-02-27.md + pnpm test:run src/audit/phase0LiveBaseline.test.ts src/audit/phase0BaselineSummary.test.ts + pnpm typecheck passing"
},
"phase0-live-baseline-capture-tooling": {
"status": "completed",
"date": "2026-02-27",
"updated": "2026-02-27",
"summary": "Added a dedicated live phase-0 baseline capture flow that reads audit logs, filters run/reaction telemetry, excludes probe sessions, anonymizes session/sender/request IDs, and writes sample + summary artifacts for operational refreshes.",
"summary": "Added a dedicated live phase-0 baseline capture flow that reads audit logs, filters run/reaction telemetry, excludes probe sessions, anonymizes session/sender/request IDs, and writes sample + summary artifacts for operational refreshes across both channel-origin and gateway-origin windows.",
"files_modified": [
"src/audit/phase0LiveBaseline.ts",
"src/audit/phase0LiveBaseline.test.ts",
@@ -99,9 +102,30 @@
"docs/plans/artifacts/phase0_baseline_live_2026-02-27.jsonl",
"docs/plans/artifacts/phase0_baseline_live_2026-02-27.md",
"docs/plans/artifacts/phase0_baseline_live_2026-02-27.json",
"docs/plans/artifacts/phase0_baseline_live_gateway_2026-02-27.jsonl",
"docs/plans/artifacts/phase0_baseline_live_gateway_2026-02-27.md",
"docs/plans/artifacts/phase0_baseline_live_gateway_2026-02-27.json",
"docs/plans/state.json"
],
"test_status": "pnpm audit:phase0-baseline:live + pnpm test:run src/audit/phase0LiveBaseline.test.ts src/audit/phase0BaselineSummary.test.ts + pnpm typecheck passing"
"test_status": "pnpm audit:phase0-baseline:live + node --import tsx/esm scripts/capture-phase0-live-baseline.ts --audit ~/.local/share/flynn/audit.log --source gateway --since 1772178440693 --until 1772178442694 --sample-out docs/plans/artifacts/phase0_baseline_live_gateway_2026-02-27.jsonl --summary-json-out docs/plans/artifacts/phase0_baseline_live_gateway_2026-02-27.json --summary-md-out docs/plans/artifacts/phase0_baseline_live_gateway_2026-02-27.md + pnpm test:run src/audit/phase0LiveBaseline.test.ts src/audit/phase0BaselineSummary.test.ts + pnpm typecheck passing"
},
"phase0-live-baseline-gateway-window": {
"status": "completed",
"date": "2026-02-27",
"updated": "2026-02-27",
"summary": "Captured and committed a gateway-origin live phase-0 baseline window with explicit cancel-path coverage (`run.cancel`, `run.state.cancel_requested`, `run.state.cancelled`) plus a paired completion sample, producing anonymized `phase0_baseline_live_gateway_2026-02-27.*` artifacts.",
"files_modified": [
"README.md",
"docs/api/PROTOCOL.md",
"docs/architecture/AGENT_DIAGRAM.md",
"docs/architecture/GATEWAY_SESSIONS_AND_QUEUE.md",
"docs/plans/2026-02-25-phase0-instrumentation-ticket-checklist.md",
"docs/plans/artifacts/phase0_baseline_live_gateway_2026-02-27.jsonl",
"docs/plans/artifacts/phase0_baseline_live_gateway_2026-02-27.md",
"docs/plans/artifacts/phase0_baseline_live_gateway_2026-02-27.json",
"docs/plans/state.json"
],
"test_status": "node --import tsx/esm scripts/capture-phase0-live-baseline.ts --audit ~/.local/share/flynn/audit.log --source gateway --since 1772178440693 --until 1772178442694 --sample-out docs/plans/artifacts/phase0_baseline_live_gateway_2026-02-27.jsonl --summary-json-out docs/plans/artifacts/phase0_baseline_live_gateway_2026-02-27.json --summary-md-out docs/plans/artifacts/phase0_baseline_live_gateway_2026-02-27.md + pnpm test:run src/audit/phase0LiveBaseline.test.ts src/audit/phase0BaselineSummary.test.ts + pnpm typecheck passing"
},
"phase0-instrumentation-ticket-checklist": {
"status": "completed",
@@ -7288,8 +7312,8 @@
"deeper_surfaces_phase0_ticket_02": "completed — gateway + daemon routing emit run lifecycle/cancel telemetry and reaction match/skip audit events with filter summaries and cancellation latency, plus focused tests",
"deeper_surfaces_phase0_ticket_03": "completed — gateway metrics now track run-state outcomes, cancel latency samples, and reaction decision counters with routing/gateway emitters",
"deeper_surfaces_phase0_ticket_04": "completed — added phase-0 baseline summary tooling for run outcomes, cancel latency, and reaction decisions with markdown/json CLI output",
"deeper_surfaces_phase0_ticket_05": "completed — documented phase-0 telemetry fields/workflow, refreshed architecture/protocol docs, and generated anonymized live baseline artifacts from real channel audit traffic (probe-only artifact workflow superseded)",
"next_up": "Capture a gateway-origin live phase-0 baseline sample (including run.cancel/cancelled paths) and append as a second live artifact window alongside the channel sample",
"deeper_surfaces_phase0_ticket_05": "completed — documented phase-0 telemetry fields/workflow, refreshed architecture/protocol docs, and generated anonymized live baseline artifacts for both channel-origin and gateway-origin traffic (including cancel-path coverage)",
"next_up": "Phase-0 baseline windows now cover channel and gateway sources; keep both artifact windows refreshed on cadence before additional run-control/reaction semantic changes.",
"pi_embedded_canary_spike": "completed — added optional pi_embedded backend adapter, canary-safe no-tools routing guard, backend success/fallback latency audit events, and docs/diagram updates while native remains default",
"pi_embedded_evaluation_phase": "completed — final decision rollback (applied in runtime config): Window A failed latency/fallback gates (p50 +259ms, p95 +5695ms, fallback 25%, categories: pi_module_interface/empty_assistant_text); Window B remained sample-insufficient; controlled probes verified guard coverage (pi_no_tools_mode/capability_query/attachments_present each hit once)",
"pi_embedded_manual_mode": "completed — added persisted runtime backend controls for manual Pi activation/deactivation (`/runtime` preferred, `/backend` alias; `status`, `activate pi`, `deactivate pi`, `use config`) while keeping config-driven default routing",