docs: document native audio support across README, CHANGELOG, config, and planning docs
- README: add audio.transcribe to tool list, update media pipeline description, add Native Audio Support and Audio Transcription config sections, add supports_audio per-tier override example - SOUL.md: add audio.transcribe to available tools list - CHANGELOG: add native audio support and audio.transcribe tool entries - config/default.yaml: add commented audio config section, supports_audio hint - INTEGRATIONS.md: expand audio section with native passthrough, capabilities, smart routing, AudioSource type, token estimation, audio.transcribe tool - STRUCTURE.md: add capabilities.ts and audio-transcribe.ts to key file listings - ARCHITECTURE.md: update data flow step 5 to describe smart audio routing
This commit is contained in:
William Valentin
@@ -161,7 +161,7 @@
|
||||
2. Adapter calls `onMessage()` callback → `ChannelRegistry.handleInbound()` routes to `MessageHandler`
|
||||
3. `createMessageRouter()` resolves agent config via `AgentRouter.resolve(channel, senderId)`
|
||||
4. `getOrCreateAgent()` creates/retrieves `AgentOrchestrator` for the session (cached by `channel:sender:agentConfig`)
|
||||
5. Audio attachments transcribed if present
|
||||
5. Audio routing: `supportsAudioInput()` checks provider capability — native audio passed through for Gemini/OpenAI/GitHub, transcribed via Whisper for others
|
||||
6. `orchestrator.process()` → injects memory context → checks compaction → delegates to `NativeAgent.process()`
|
||||
7. `NativeAgent.toolLoop()` → sends to `ModelRouter.chat()` → model returns response or tool calls
|
||||
8. If tool calls: `ToolExecutor.execute()` → policy check → hook check → tool execution → loop back to model
|
||||
|
||||
@@ -234,6 +234,22 @@ All adapters implement `ChannelAdapter` interface (`src/channels/types.ts`): `co
|
||||
- Supported formats: OGG, MP3, WAV, WebM, MP4, M4A
|
||||
- Integration: Auto-transcribes audio attachments from channels before model processing
|
||||
|
||||
**Native Audio Passthrough:**
|
||||
- Implementation: `src/models/capabilities.ts`, `src/daemon/routing.ts`
|
||||
- Capability check: `supportsAudioInput(provider, model, override?)` determines if a model can process raw audio
|
||||
- Audio-capable providers: Gemini (`inlineData`), OpenAI (`input_audio`), GitHub (`input_audio`)
|
||||
- Non-audio providers: Anthropic, Bedrock, Ollama, llama.cpp (fall back to Whisper transcription)
|
||||
- Config override: `supports_audio: true/false` per model tier overrides auto-detection
|
||||
- Smart routing: `createMessageRouter()` checks capability, passes raw `AudioSource` for capable models or transcribes via Whisper for others
|
||||
- Audio content types: `AudioSource` (`{ type: 'audio', data: string, mimeType: string }`) in `src/models/types.ts`
|
||||
- Token estimation: `estimateAudioTokens()` in `src/context/tokens.ts` (base64 length -> bytes -> duration at 16kbps -> tokens at 32/sec)
|
||||
|
||||
**Agent Tool: audio.transcribe:**
|
||||
- Implementation: `src/tools/builtin/audio-transcribe.ts`
|
||||
- Transcribes audio files on-demand via the configured Whisper-compatible endpoint
|
||||
- Input: file path or base64 data with MIME type
|
||||
- Output: transcribed text
|
||||
|
||||
## MCP (Model Context Protocol)
|
||||
|
||||
**MCP Client:**
|
||||
|
||||
@@ -150,7 +150,7 @@ flynn/
|
||||
**`src/models/`:**
|
||||
- Purpose: LLM provider client implementations and tier-based routing
|
||||
- Contains: Provider clients, `ModelRouter`, retry logic, cost estimation, media helpers
|
||||
- Key files: `src/models/types.ts` (core interfaces), `src/models/router.ts`, `src/models/anthropic.ts`, `src/models/openai.ts`, `src/models/gemini.ts`, `src/models/bedrock.ts`, `src/models/github.ts`, `src/models/retry.ts`, `src/models/costs.ts`, `src/models/media.ts`
|
||||
- Key files: `src/models/types.ts` (core interfaces), `src/models/router.ts`, `src/models/anthropic.ts`, `src/models/openai.ts`, `src/models/gemini.ts`, `src/models/bedrock.ts`, `src/models/github.ts`, `src/models/retry.ts`, `src/models/costs.ts`, `src/models/media.ts`, `src/models/capabilities.ts`
|
||||
|
||||
**`src/models/local/`:**
|
||||
- Purpose: Local model provider clients
|
||||
@@ -185,7 +185,7 @@ flynn/
|
||||
**`src/tools/builtin/`:**
|
||||
- Purpose: Built-in tool implementations shipped with Flynn
|
||||
- Contains: Shell exec, file operations, web fetch, memory ops, web search, media send, image analysis, session management, agent listing, cross-channel messaging, cron management
|
||||
- Key files: `src/tools/builtin/shell.ts`, `src/tools/builtin/file-read.ts`, `src/tools/builtin/file-write.ts`, `src/tools/builtin/file-edit.ts`, `src/tools/builtin/file-patch.ts`, `src/tools/builtin/file-list.ts`, `src/tools/builtin/web-fetch.ts`, `src/tools/builtin/web-search.ts`, `src/tools/builtin/memory-read.ts`, `src/tools/builtin/memory-write.ts`, `src/tools/builtin/memory-search.ts`, `src/tools/builtin/media-send.ts`, `src/tools/builtin/image-analyze.ts`, `src/tools/builtin/system-info.ts`, `src/tools/builtin/sessions.ts`, `src/tools/builtin/agents-list.ts`, `src/tools/builtin/message-send.ts`, `src/tools/builtin/cron.ts`
|
||||
- Key files: `src/tools/builtin/shell.ts`, `src/tools/builtin/file-read.ts`, `src/tools/builtin/file-write.ts`, `src/tools/builtin/file-edit.ts`, `src/tools/builtin/file-patch.ts`, `src/tools/builtin/file-list.ts`, `src/tools/builtin/web-fetch.ts`, `src/tools/builtin/web-search.ts`, `src/tools/builtin/memory-read.ts`, `src/tools/builtin/memory-write.ts`, `src/tools/builtin/memory-search.ts`, `src/tools/builtin/media-send.ts`, `src/tools/builtin/image-analyze.ts`, `src/tools/builtin/audio-transcribe.ts`, `src/tools/builtin/system-info.ts`, `src/tools/builtin/sessions.ts`, `src/tools/builtin/agents-list.ts`, `src/tools/builtin/message-send.ts`, `src/tools/builtin/cron.ts`
|
||||
|
||||
**`src/tools/builtin/browser/`:**
|
||||
- Purpose: Puppeteer-based browser automation tools
|
||||
|
||||
@@ -6,6 +6,15 @@ All notable changes to Flynn are documented in this file.
|
||||
|
||||
### Added
|
||||
|
||||
- **Native Audio Support** -- Smart routing for voice messages: audio-capable models
|
||||
(Gemini, OpenAI, GitHub) receive raw audio directly via `AudioSource` content parts;
|
||||
non-audio models (Anthropic, Bedrock, Ollama, llama.cpp) get Whisper transcription
|
||||
fallback. `supportsAudioInput()` capability check with per-model `supports_audio`
|
||||
config override. Audio token estimation (base64 -> bytes -> duration -> tokens at
|
||||
32 tokens/sec). 38 new tests (18 capabilities + 15 media + 5 token estimation).
|
||||
- **Agent Tool: audio.transcribe** -- Transcribe audio files via a Whisper-compatible
|
||||
API endpoint. Configurable via `audio.transcription_endpoint`, supports OGG, MP3,
|
||||
WAV, WebM, MP4, M4A formats.
|
||||
- **xAI (Grok) Provider** -- xAI as OpenAI-compatible model provider with
|
||||
`provider: xai` config. Supports grok-3, grok-3-mini, grok-2, grok-2-mini,
|
||||
grok-3-fast. Uses `XAI_API_KEY` env var or config `api_key`.
|
||||
|
||||
@@ -12,10 +12,10 @@ Self-hosted personal AI assistant with Telegram and Terminal interfaces.
|
||||
- **Session Persistence**: SQLite-backed conversation history
|
||||
- **Fallback Chains**: Automatic failover when primary model fails
|
||||
- **Hook Engine**: Confirmation system for sensitive operations
|
||||
- **Tool Framework**: Shell, file, file patch, web-fetch, web-search, browser control, image analysis, media send, system info
|
||||
- **Tool Framework**: Shell, file, file patch, web-fetch, web-search, browser control, image analysis, media send, audio transcribe, system info
|
||||
- **Docker Sandboxing**: Per-session container isolation for tool execution
|
||||
- **Multi-Agent Routing**: Config-driven agent selection per sender/channel with tool profiles
|
||||
- **Media Pipeline**: Image analysis, outbound attachments, audio transcription across all channels
|
||||
- **Media Pipeline**: Image analysis, outbound attachments, audio transcription and native audio passthrough across all channels
|
||||
- **Session Transfer**: Move conversations between frontends
|
||||
- **CLI**: Full command-line interface (`flynn start`, `send`, `doctor`, `completion`, etc.)
|
||||
- **Shell Completion**: Auto-generated completions for bash, zsh, and fish with `--install` flag
|
||||
@@ -143,6 +143,48 @@ models:
|
||||
local: { provider: ollama, model: qwen2.5:14b }
|
||||
```
|
||||
|
||||
### Native Audio Support
|
||||
|
||||
Voice messages from channels can be handled in two ways:
|
||||
|
||||
1. **Native passthrough** -- Audio sent directly to models that support audio input (Gemini, OpenAI, GitHub). No transcription step needed.
|
||||
2. **Whisper transcription** -- Audio transcribed to text via a Whisper-compatible API, then sent as text to models that don't support audio input (Anthropic, Bedrock, Ollama, llama.cpp).
|
||||
|
||||
Flynn automatically routes based on the model's capabilities. You can override this per-tier:
|
||||
|
||||
```yaml
|
||||
models:
|
||||
default:
|
||||
provider: gemini
|
||||
model: gemini-2.0-flash
|
||||
supports_audio: true # Force native audio (auto-detected for known providers)
|
||||
fast:
|
||||
provider: anthropic
|
||||
model: claude-sonnet-4
|
||||
supports_audio: false # Force transcription (default for Anthropic)
|
||||
```
|
||||
|
||||
### Audio Transcription
|
||||
|
||||
Configure a Whisper-compatible endpoint for models that don't support native audio:
|
||||
|
||||
```yaml
|
||||
audio:
|
||||
transcription_endpoint: "http://localhost:8080/v1/audio/transcriptions"
|
||||
transcription_api_key: "${WHISPER_API_KEY}" # Optional Bearer token
|
||||
transcription_model: "whisper-1" # Model name (default: whisper-1)
|
||||
transcription_provider: "openai" # Provider format: openai (default)
|
||||
```
|
||||
|
||||
| Field | Required | Description |
|
||||
|-------|----------|-------------|
|
||||
| `transcription_endpoint` | yes | Whisper-compatible API endpoint |
|
||||
| `transcription_api_key` | no | Bearer token for authentication |
|
||||
| `transcription_model` | no | Model name sent in the request (default: `whisper-1`) |
|
||||
| `transcription_provider` | no | API format: `openai` (default) |
|
||||
|
||||
Without an `audio` config, voice messages from non-audio-capable models are silently skipped.
|
||||
|
||||
## Telegram Commands
|
||||
|
||||
| Command | Description |
|
||||
@@ -726,12 +768,12 @@ src/
|
||||
├── hooks/ # Confirmation engine
|
||||
├── mcp/ # MCP tool server integration
|
||||
├── memory/ # Persistent memory store + vector search
|
||||
├── models/ # Model providers + router + media pipeline
|
||||
├── models/ # Model providers + router + media pipeline + audio capabilities
|
||||
├── prompt/ # System prompt templating (auto-injects current date/time)
|
||||
├── sandbox/ # Docker sandboxing
|
||||
├── session/ # SQLite persistence
|
||||
├── skills/ # Skill packages
|
||||
├── tools/ # Builtin tools (shell, file, web, browser, process, media, system.info)
|
||||
├── tools/ # Builtin tools (shell, file, web, browser, process, media, audio, system.info)
|
||||
└── automation/ # Cron scheduler, webhooks, heartbeat monitor, Gmail watcher
|
||||
```
|
||||
|
||||
|
||||
@@ -55,7 +55,7 @@ You have tools for interacting with your operator's system:
|
||||
- **process.start / process.status / process.output / process.kill / process.list** -- Manage background processes.
|
||||
- **message.send** -- Send messages to other channels (Telegram, Discord, etc.).
|
||||
|
||||
Additional tools (image.analyze, media.send, browser.*, gmail.*, calendar.*, sessions.*, agents.list) may be available depending on configuration. Check your tool definitions if unsure.
|
||||
Additional tools (image.analyze, media.send, audio.transcribe, browser.*, gmail.*, calendar.*, sessions.*, agents.list) may be available depending on configuration. Check your tool definitions if unsure.
|
||||
|
||||
## Tool Usage Rules
|
||||
|
||||
|
||||
@@ -39,6 +39,7 @@ models:
|
||||
default:
|
||||
provider: anthropic
|
||||
model: claude-sonnet-4-20250514
|
||||
# supports_audio: false # Override native audio detection per tier
|
||||
local:
|
||||
provider: ollama
|
||||
model: glm-4.7-flash
|
||||
@@ -117,3 +118,14 @@ hooks:
|
||||
# peer: "123456789"
|
||||
# failure_threshold: 2
|
||||
# disk_threshold_mb: 100
|
||||
|
||||
# ── Audio ────────────────────────────────────────────────────────────
|
||||
# Configure a Whisper-compatible endpoint for audio transcription.
|
||||
# Models that support native audio input (Gemini, OpenAI, GitHub) will
|
||||
# receive raw audio directly; others fall back to this endpoint.
|
||||
|
||||
# audio:
|
||||
# transcription_endpoint: "http://localhost:8080/v1/audio/transcriptions"
|
||||
# transcription_api_key: "${WHISPER_API_KEY}"
|
||||
# transcription_model: "whisper-1"
|
||||
# transcription_provider: "openai"
|
||||
|
||||
Reference in New Issue
Block a user