flynn/docs/plans/2026-02-25-deeper-end-user-surfaces-and-integrated-behavior-stack-plan.md

Flynn Deeper End-User Surfaces + Integrated Behavior Stack Plan

Date: 2026-02-25
Status: proposed roadmap
Scope: deepen assistant "product feel" (behavior semantics + user-facing surfaces) without rewriting Flynn core architecture

Summary

This plan adopts a balanced hybrid strategy:

1. Improve behavior semantics first where correctness risk is highest (interrupt/cancel/run control).
2. In parallel, ship selective deeper user surfaces (companion, canvas persistence, voice continuity).
3. Land each slice with explicit observability gates so rollout decisions are data-driven.

Why This Plan

Flynn already has strong foundations:

• Queue + session orchestration: src/gateway/lane-queue.ts, src/gateway/session-bridge.ts
• Multi-path routing and backend fallback: src/daemon/routing.ts
• Companion RPC foundation: src/companion/runtimeClient.ts, src/gateway/protocol.ts
• Canvas API baseline: src/gateway/handlers/canvas.ts, src/gateway/canvas-store.ts
• Voice in/out primitives: src/models/media.ts, src/models/tts.ts
• Reactions baseline: src/automation/reactions.ts

Largest remaining gap vs OpenClaw-like "assistant feel" is integration behavior across those systems, not missing foundational architecture.

Goals and Success Criteria

1. Deterministic active-run control under bursty traffic
2. Rich, safe proactive behavior stack
3. Durable end-user surfaces for companion/canvas/voice
4. Measurable reliability improvements across canary phases

Quantitative success gates:

1. Cancel-to-ack p95 <= 500ms on gateway sessions.
2. Duplicate assistant responses caused by run preemption: 0 in integration tests.
3. Reaction false-positive rate <= 3% in canary logs.
4. Companion reconnect success >= 99% in soak tests.
5. Canvas artifact persistence survives daemon restart in integration tests.
6. Voice failures degrade to text-only replies with no dropped responses.

Out of Scope

1. Full native macOS/iOS/Android app suite in this phase set.
2. Broad protocol redesign or protocol-version breaking changes.
3. Pi backend expansion (kept separate from this roadmap until re-approval).

Workstreams and Complexity

Workstream	Complexity	Main Risk
Run-control semantics unification	High	race conditions and cancellation ordering
Reactions + proactive behavior v2	Medium-High	noisy or looping automation
Companion + canvas + voice deepening	High	cross-surface consistency and restart behavior
Rollout hardening + observability	Medium	incomplete canary signals

Phase 0 - Baseline Instrumentation and Guardrails

Duration: 3-5 days

Execution checklist:

• docs/plans/2026-02-25-phase0-instrumentation-ticket-checklist.md

Deliverables

1. Add baseline event and metric instrumentation for:
   • run-state transitions,
   • cancellation path timings,
   • reaction match/skip reasons,
   • surface delivery outcomes.
2. Define canary gate calculator inputs for each later phase.

Files

1. src/audit/types.ts
2. src/audit/logger.ts
3. src/gateway/metrics.ts
4. docs/api/PROTOCOL.md (event semantics if needed)
5. docs/plans/state.json

Acceptance

1. Baseline report generated before behavior changes.
2. No runtime behavior changes in this phase, only observability.

Phase 1 - Run-Control Semantics + Gateway UX Signals

Duration: 2-3 weeks

Objectives

1. Enforce "latest wins" semantics when queue policy is interrupt.
2. Align cancellation behavior between gateway and channel router paths.
3. Expose user-visible run lifecycle state in gateway events/UI.

Implementation

1. Queue semantics hardening:
   • src/gateway/lane-queue.ts
   • ensure queued + active behavior rules are explicit and testable.
2. Active run cancellation wiring:
   • src/gateway/handlers/agent.ts
   • src/gateway/session-bridge.ts
   • src/daemon/routing.ts (activeRuns parity behavior).
3. Event surface:
   • add additive run_state event in src/gateway/protocol.ts
   • consume/render in src/gateway/ui/pages/chat.js.

Test Plan

1. src/gateway/lane-queue.test.ts: preemption ordering, overflow with interrupt, debounce edge cases.
2. src/gateway/handlers/agent.test.ts: interrupt + active cancel + queued supersede flows.
3. src/daemon/routing.test.ts: channel-path cancellation parity.
4. src/gateway/ui/pages/chat.test.ts: run_state rendering and transitions.

Acceptance

1. Cancel-to-ack p95 <= 500ms.
2. Zero duplicate final responses in integration suite.
3. Backward compatibility for clients ignoring run_state.

Phase 2 - Reactions and Proactive Behavior Stack V2

Duration: 2 weeks

Objectives

1. Replace first-match reaction behavior with deterministic priority + cooldown semantics.
2. Keep announce delivery safe and auditable.
3. Prevent recursion/looping behavior.

Config/API Additions (backward-compatible)

Extend automation.reactions[] in src/config/schema.ts with:

1. priority (number, default 100)
2. cooldown_ms (number, default 0)
3. stop_on_match (boolean, default true)

Existing fields remain valid and unchanged.

Implementation

1. Reaction engine expansion:
   • src/automation/reactions.ts (or split reactionEngine.ts if needed).
2. Routing integration:
   • src/daemon/routing.ts deterministic reaction resolution.
3. Delivery consistency:
   • src/automation/cron.ts
   • src/automation/webhooks.ts
   • preserve delivery_mode semantics and audit metadata.

Test Plan

1. src/automation/reactions.test.ts:
   • priority conflict resolution,
   • cooldown suppression,
   • metadata and template rendering.
2. src/daemon/routing.test.ts:
   • reaction trigger integration and command-path exclusion.
3. src/automation/cron.test.ts / src/automation/webhooks.test.ts:
   • announce/isolation metadata correctness.

Acceptance

1. False-positive match rate <= 3% in canary.
2. No reaction recursion loops.
3. Deterministic rule selection under overlap.

Phase 3 - Deeper Surfaces: Companion, Canvas Durability, Voice Continuity

Duration: 3-4 weeks

Objectives

1. Upgrade companion from heartbeat-only utility to reliable daily-use surface.
2. Make canvas artifacts durable across restart.
3. Improve voice continuity behavior around cancellation and fallbacks.

Implementation

1. Companion hardening:
   • src/cli/companion.ts
   • src/companion/runtimeClient.ts
   • src/gateway/handlers/node.ts
   • focus on reconnect and subscription resilience.
2. Canvas persistence:
   • src/gateway/canvas-store.ts (durable backing instead of in-memory only)
   • src/gateway/handlers/canvas.ts
   • UI rendering/inspection in src/gateway/ui/pages/chat.js (or dedicated canvas page).
3. Voice continuity:
   • src/daemon/routing.ts (talk-mode + cancellation + output behavior)
   • src/models/tts.ts
   • channel adapter output checks where required.

Test Plan

1. Companion integration tests for reconnect and event continuity.
2. Canvas store tests for restart durability and eviction policy.
3. Voice tests for TTS errors, fallback to text, and interrupted runs.

Acceptance

1. Companion reconnect success >= 99% in soak.
2. Canvas survives daemon restart in integration suite.
3. Voice path never drops assistant reply when TTS fails.

Phase 4 - Rollout, Hardening, and Operator Readiness

Duration: 1 week

Deliverables

1. Canary rollout plan by feature flag/surface.
2. Explicit rollback playbook.
3. Operator docs and architecture/protocol docs synchronized.

Documentation Updates (required in same PRs)

1. README.md
2. docs/api/PROTOCOL.md
3. docs/architecture/AGENT_DIAGRAM.md
4. docs/architecture/GATEWAY_SESSIONS_AND_QUEUE.md
5. docs/plans/state.json

Execution and Commit Structure

1. Branch: feature/deeper-surfaces-integrated-behavior-stack
2. Atomic commits per slice:
   • implementation + tests + docs + state.json together
3. Rebase onto main before merge.
4. Fast-forward merge only.

Model-Tier Delegation Plan for Implementation Work

1. claude-haiku-4.5:
   • mechanical schema/test/doc updates.
2. claude-sonnet-4.6:
   • default implementation tasks across queue/routing/companion/canvas.
3. claude-opus-4.6:
   • concurrency semantics review, failure-mode design, and rollout gate design.

Risks and Mitigations

1. Risk: preemption races create duplicate or orphaned replies.
   Mitigation: run-state event model + deterministic cancellation tests.
2. Risk: proactive rules become noisy.
   Mitigation: priority/cooldown/stop semantics + canary thresholds.
3. Risk: deeper surfaces drift from core behavior semantics.
   Mitigation: shared gateway protocol contracts and integration tests across surfaces.

Default Decisions Locked

1. Keep gateway protocol backward-compatible (additive only).
2. Prioritize behavior reliability before broadening platform count.
3. Use companion CLI/runtime path as first deep-surface target.
4. Keep Pi expansion out of this roadmap until separate canary re-approval.