docs(plan): record voice wake and capture gap completion
This commit is contained in:
@@ -174,8 +174,8 @@ Flynn actually has MCP support that OpenClaw doesn't emphasise — OpenClaw reli
|
|||||||
| macOS menu bar app | Full | -- | **MISSING** |
|
| macOS menu bar app | Full | -- | **MISSING** |
|
||||||
| iOS node | Full (Canvas, Voice, Camera) | -- | **MISSING** |
|
| iOS node | Full (Canvas, Voice, Camera) | -- | **MISSING** |
|
||||||
| Android node | Full (Canvas, Talk, Camera) | -- | **MISSING** |
|
| Android node | Full (Canvas, Talk, Camera) | -- | **MISSING** |
|
||||||
| Voice Wake / Talk Mode | Full (ElevenLabs) | -- | **MISSING** |
|
| Voice Wake / Talk Mode | Full (ElevenLabs) | Wake phrase + timed talk mode | **MATCH** |
|
||||||
| Camera / screen capture | Via nodes | -- | **MISSING** |
|
| Camera / screen capture | Via nodes | `screen.capture` + `camera.capture` tools | **MATCH** |
|
||||||
| Location access | Via nodes | -- | **MISSING** |
|
| Location access | Via nodes | -- | **MISSING** |
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -299,7 +299,6 @@ All five Tier 3 items implemented: Lane Queue (per-session FIFO in gateway), cre
|
|||||||
- Companion apps (macOS/iOS/Android) — massive scope
|
- Companion apps (macOS/iOS/Android) — massive scope
|
||||||
- LINE, Feishu, Mattermost — niche audience
|
- LINE, Feishu, Mattermost — niche audience
|
||||||
- Canvas/A2UI — experimental visual workspace
|
- Canvas/A2UI — experimental visual workspace
|
||||||
- Voice Wake / Talk Mode — ElevenLabs TTS integration
|
|
||||||
- Nix/Fly.io/Railway deployment — platform-specific
|
- Nix/Fly.io/Railway deployment — platform-specific
|
||||||
- OAuth subscription auth — complex
|
- OAuth subscription auth — complex
|
||||||
- Skill/plugin safety scanner — static analysis
|
- Skill/plugin safety scanner — static analysis
|
||||||
|
|||||||
@@ -62,8 +62,6 @@ A gap item is considered implemented when:
|
|||||||
- macOS menu bar app
|
- macOS menu bar app
|
||||||
- iOS node
|
- iOS node
|
||||||
- Android node
|
- Android node
|
||||||
- Voice Wake / Talk Mode
|
|
||||||
- Camera / screen capture
|
|
||||||
- Location access
|
- Location access
|
||||||
|
|
||||||
## Roadmap Overview (Milestones)
|
## Roadmap Overview (Milestones)
|
||||||
@@ -323,7 +321,7 @@ These are substantial UX/ecosystem projects or highly platform-specific; defer u
|
|||||||
|
|
||||||
- Canvas/A2UI
|
- Canvas/A2UI
|
||||||
- Companion apps (macOS/iOS/Android)
|
- Companion apps (macOS/iOS/Android)
|
||||||
- Voice wake/talk mode + camera/screen capture/location
|
- Location access
|
||||||
- Presence tracking
|
- Presence tracking
|
||||||
- Bonjour/mDNS discovery
|
- Bonjour/mDNS discovery
|
||||||
- QMD backend
|
- QMD backend
|
||||||
|
|||||||
@@ -0,0 +1,41 @@
|
|||||||
|
# Voice Wake + Capture Checklist
|
||||||
|
|
||||||
|
Date: 2026-02-16
|
||||||
|
Status: completed
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
- Implement a practical Voice Wake / Talk Mode slice.
|
||||||
|
- Implement Camera / Screen Capture via host tools.
|
||||||
|
|
||||||
|
## Completed
|
||||||
|
|
||||||
|
- Added `audio.talk_mode` config schema:
|
||||||
|
- `enabled` (default `false`)
|
||||||
|
- `wake_phrase` (default `hey flynn`)
|
||||||
|
- `timeout_ms` (default `120000`)
|
||||||
|
- `allow_manual_toggle` (default `true`)
|
||||||
|
- Wired talk mode gating in `src/daemon/routing.ts`:
|
||||||
|
- wake phrase activation with timed listen window
|
||||||
|
- manual controls: `/talk on`, `/talk off`, `/talk status`
|
||||||
|
- idle mode drops non-wake text when talk mode is enabled
|
||||||
|
- Added built-in capture tools in `src/tools/builtin/capture.ts`:
|
||||||
|
- `screen.capture`
|
||||||
|
- `camera.capture`
|
||||||
|
- returns base64 image payloads (`image/png` or `image/jpeg`)
|
||||||
|
- Registered capture tools in builtin index and tool policy runtime/coding sets.
|
||||||
|
- Updated docs:
|
||||||
|
- `README.md`
|
||||||
|
- `config/default.yaml`
|
||||||
|
- gap/roadmap docs in `docs/plans/`
|
||||||
|
|
||||||
|
## Tests
|
||||||
|
|
||||||
|
- `src/daemon/routing.test.ts` (wake phrase/talk mode behavior)
|
||||||
|
- `src/tools/builtin/capture.test.ts` (capture tool command wrapping behavior)
|
||||||
|
- `src/config/schema.test.ts` (talk mode schema defaults/overrides)
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
- `pnpm test:run src/daemon/routing.test.ts src/tools/builtin/capture.test.ts src/config/schema.test.ts src/tools/policy.test.ts`
|
||||||
|
- `pnpm typecheck`
|
||||||
+35
-2
@@ -581,6 +581,39 @@
|
|||||||
],
|
],
|
||||||
"test_status": "pnpm test:run src/channels/bluebubbles/adapter.test.ts src/config/schema.test.ts src/gateway/handlers/services.test.ts + pnpm typecheck passing"
|
"test_status": "pnpm test:run src/channels/bluebubbles/adapter.test.ts src/config/schema.test.ts src/gateway/handlers/services.test.ts + pnpm typecheck passing"
|
||||||
},
|
},
|
||||||
|
"voice-wake-talk-mode": {
|
||||||
|
"file": "2026-02-16-voice-wake-and-capture-checklist.md",
|
||||||
|
"status": "completed",
|
||||||
|
"date": "2026-02-16",
|
||||||
|
"updated": "2026-02-16",
|
||||||
|
"summary": "Added configurable wake-phrase talk mode (`audio.talk_mode`) with timed listen windows and `/talk on|off|status` controls in the message router.",
|
||||||
|
"files_modified": [
|
||||||
|
"src/config/schema.ts",
|
||||||
|
"src/config/schema.test.ts",
|
||||||
|
"src/daemon/routing.ts",
|
||||||
|
"src/daemon/routing.test.ts",
|
||||||
|
"README.md",
|
||||||
|
"config/default.yaml"
|
||||||
|
],
|
||||||
|
"test_status": "pnpm test:run src/daemon/routing.test.ts src/config/schema.test.ts + pnpm typecheck passing"
|
||||||
|
},
|
||||||
|
"camera-screen-capture": {
|
||||||
|
"file": "2026-02-16-voice-wake-and-capture-checklist.md",
|
||||||
|
"status": "completed",
|
||||||
|
"date": "2026-02-16",
|
||||||
|
"updated": "2026-02-16",
|
||||||
|
"summary": "Added host capture tools (`screen.capture`, `camera.capture`) with platform command wrappers and base64 image payload output for camera/screen workflows.",
|
||||||
|
"files_created": [
|
||||||
|
"src/tools/builtin/capture.ts",
|
||||||
|
"src/tools/builtin/capture.test.ts"
|
||||||
|
],
|
||||||
|
"files_modified": [
|
||||||
|
"src/tools/builtin/index.ts",
|
||||||
|
"src/tools/policy.ts",
|
||||||
|
"README.md"
|
||||||
|
],
|
||||||
|
"test_status": "pnpm test:run src/tools/builtin/capture.test.ts src/tools/policy.test.ts + pnpm typecheck passing"
|
||||||
|
},
|
||||||
"openclaw-style-personal-agent-without-openclaw-risks": {
|
"openclaw-style-personal-agent-without-openclaw-risks": {
|
||||||
"file": "2026-02-14-openclaw-style-personal-agent-without-openclaw-risks-plan.md",
|
"file": "2026-02-14-openclaw-style-personal-agent-without-openclaw-risks-plan.md",
|
||||||
"status": "completed",
|
"status": "completed",
|
||||||
@@ -2903,12 +2936,12 @@
|
|||||||
"tier2_completion": "4/4 (100%) — inbound webhooks, vector memory search, Dockerfile, heartbeat monitor",
|
"tier2_completion": "4/4 (100%) — inbound webhooks, vector memory search, Dockerfile, heartbeat monitor",
|
||||||
"tier3_completion": "5/5 (100%) — lane queue, credential redaction, web UI token dashboard, xAI (Grok) provider, Voyage AI embeddings",
|
"tier3_completion": "5/5 (100%) — lane queue, credential redaction, web UI token dashboard, xAI (Grok) provider, Voyage AI embeddings",
|
||||||
"tier4_completion": "4/4 (100%) — gateway lock, shell completion, Tailscale Serve/Funnel, DM pairing codes",
|
"tier4_completion": "4/4 (100%) — gateway lock, shell completion, Tailscale Serve/Funnel, DM pairing codes",
|
||||||
"feature_gap_scorecard": "113/128 match (88%), 0 partial (0%), 15 missing (12%)",
|
"feature_gap_scorecard": "115/128 match (90%), 0 partial (0%), 13 missing (10%)",
|
||||||
"operator_dx_milestone": "Phase 3 (Live Ops Dashboard): 2/2 plans complete — milestone done",
|
"operator_dx_milestone": "Phase 3 (Live Ops Dashboard): 2/2 plans complete — milestone done",
|
||||||
"gmail_auth_cli": "flynn gmail-auth command implemented with OAuth2 flow, doctor check, config routed to Telegram",
|
"gmail_auth_cli": "flynn gmail-auth command implemented with OAuth2 flow, doctor check, config routed to Telegram",
|
||||||
"native_audio_support": "completed — smart routing for native audio (Gemini/OpenAI/GitHub) vs Whisper transcription fallback",
|
"native_audio_support": "completed — smart routing for native audio (Gemini/OpenAI/GitHub) vs Whisper transcription fallback",
|
||||||
"remaining_phases_completion": "Phase 1: 3/3 (100%) — context levels, command registry, memory structure. Phase 2: 3/3 (100%) — component registry, confidence routing, history index. Phase 3: 2/2 (100%) — adaptive memory/compaction, truthfulness/autonomy hardening",
|
"remaining_phases_completion": "Phase 1: 3/3 (100%) — context levels, command registry, memory structure. Phase 2: 3/3 (100%) — component registry, confidence routing, history index. Phase 3: 2/2 (100%) — adaptive memory/compaction, truthfulness/autonomy hardening",
|
||||||
"next_up": "OpenClaw gap: Zalo channel adapter (open next scoped implementation checklist)"
|
"next_up": "OpenClaw gap: Location access (open next scoped implementation checklist)"
|
||||||
},
|
},
|
||||||
"soul_md_and_cron_create": {
|
"soul_md_and_cron_create": {
|
||||||
"date": "2026-02-11",
|
"date": "2026-02-11",
|
||||||
|
|||||||
Reference in New Issue
Block a user