William Valentin
1900 lines
74 KiB
Markdown
1900 lines
74 KiB
Markdown
# Flynn
|
|
|
|
Self-hosted personal AI assistant with Telegram and Terminal interfaces.
|
|
|
|
## Features
|
|
|
|
- **Multi-Frontend**: Telegram bot + Terminal UI (minimal & fullscreen modes) + Web UI dashboard
|
|
- **Multi-Model**: Anthropic Claude, OpenAI, GitHub Copilot, Gemini, Bedrock, Zhipu AI (GLM), xAI (Grok), Ollama, llama.cpp with intelligent routing
|
|
- **Multi-Channel**: Telegram, Discord, Slack, WhatsApp, Matrix, Signal, Mattermost, Microsoft Teams, Google Chat, LINE, Feishu/Lark, Zalo, and iMessage (BlueBubbles) with unified adapter interface
|
|
- **Web Dashboard**: SPA control panel with health monitoring, chat, session browser, usage stats, and settings editor
|
|
- **Model Switching**: Switch between cloud/local models on demand
|
|
- **Session Persistence**: SQLite-backed conversation history
|
|
- **Fallback Chains**: Automatic failover when primary model fails
|
|
- **Hook Engine**: Confirmation system for sensitive operations
|
|
- **Tool Framework**: Shell, file, file patch, web-fetch, web-search, browser control, image analysis, media send, audio transcribe, system info
|
|
- **Docker Sandboxing**: Per-session container isolation for tool execution
|
|
- **Multi-Agent Routing**: Config-driven agent selection per sender/channel with tool profiles
|
|
- **Media Pipeline**: Image analysis, outbound attachments, audio transcription and native audio passthrough across all channels
|
|
- **Talk Mode (Wake Phrase)**: Optional wake-phrase gating (`audio.talk_mode`) with timed conversation windows
|
|
- **Capture Tools**: `screen.capture` and `camera.capture` tools for host capture workflows
|
|
- **Session Transfer**: Move conversations between frontends
|
|
- **CLI**: Full command-line interface (`flynn start`, `send`, `doctor`, `completion`, etc.)
|
|
- **Optional Pi Embedded Backend**: Canary-only in-process Pi runtime path (`pi_embedded`) with native fallback
|
|
- **Shell Completion**: Auto-generated completions for bash, zsh, and fish with `--install` flag
|
|
- **Cron Scheduling**: Automated messages on cron schedules with output routing
|
|
- **Daily Briefing Automation**: Optional built-in morning briefing preset (calendar + inbox + tasks summary prompt)
|
|
- **Inbound Webhooks**: HTTP endpoints that trigger agent processing with HMAC auth and template rendering
|
|
- **Heartbeat Monitor**: Periodic health checks (gateway, model, channels, memory, disk) with failure notifications
|
|
- **Scheduled Backups**: Interval- or cron-based snapshot backups with optional startup run
|
|
- **MinIO File Sharing Tool**: Upload a local file and return a temporary MinIO share link via `minio.share`
|
|
- **MinIO Knowledge Ingestion Tool**: Pull text-like objects from MinIO into memory namespaces via `minio.ingest`
|
|
- **Kubernetes Homelab Tools**: Inspect pods/deployments and fetch logs via `k8s.pods`, `k8s.deployments`, `k8s.logs`
|
|
- **Gmail Pub/Sub Watcher**: Monitor Gmail inbox via Google Cloud Pub/Sub push notifications with polling fallback
|
|
- **Vector Memory Search**: Hybrid keyword + semantic search with embeddings (OpenAI, Gemini, Ollama, llama.cpp, Voyage AI)
|
|
- **Docker Deployment**: Multi-stage Dockerfile and docker-compose.yml for production containers
|
|
- **Health Diagnostics**: `flynn doctor` validates config, connectivity, and system state
|
|
- **MCP Integration**: External tool servers via Model Context Protocol
|
|
- **Skills System**: Extensible capability packages (bundled, managed, workspace tiers)
|
|
- **Gateway Lock**: Single-client mode — reject additional WebSocket connections when one is active
|
|
- **Tailscale Serve**: Auto-expose gateway via Tailscale Serve on daemon start with lifecycle management
|
|
- **DM Pairing Codes**: Allow unknown senders to pair with the bot via time-limited codes across all channels, with SQLite-backed persistence across restarts
|
|
- **Lane Queue**: Per-session FIFO queue serializes concurrent gateway requests
|
|
- **Node Capability Negotiation**: Optional companion-node registration and capability discovery over gateway RPC
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
# Install flynn CLI command (build + global link)
|
|
make install
|
|
|
|
# Copy and configure
|
|
cp config/default.yaml ~/.config/flynn/config.yaml
|
|
# Edit config with your API keys and Telegram bot token
|
|
|
|
# Optional: regenerate/check derived config profiles
|
|
pnpm config:profiles:generate
|
|
pnpm config:profiles:check
|
|
|
|
# Run
|
|
flynn start
|
|
|
|
# Or run without building
|
|
pnpm start
|
|
```
|
|
|
|
## CLI Commands
|
|
|
|
Flynn provides a full CLI via the `flynn` binary (or `npx tsx src/cli/index.ts` during development):
|
|
|
|
| Command | Description |
|
|
|---------|-------------|
|
|
| `flynn start` | Start the Flynn daemon (Telegram, WebChat, cron, Gmail watcher) |
|
|
| `flynn tui` | Launch the interactive terminal UI |
|
|
| `flynn send <message>` | Send a one-shot message and print the response |
|
|
| `flynn sessions` | List active sessions |
|
|
| `flynn doctor` | Validate config and check system health |
|
|
| `flynn config` | Show resolved configuration (secrets redacted) |
|
|
| `flynn backup` | Create a snapshot backup and optionally upload to MinIO |
|
|
| `flynn completion <shell>` | Generate shell completions (bash, zsh, fish) |
|
|
| `flynn setup` | Interactive setup wizard |
|
|
| `flynn onboard` | Guided onboarding alias for setup wizard |
|
|
| `flynn gmail-auth` | Authenticate with Gmail via OAuth2 |
|
|
| `flynn gcal-auth` | Authenticate with Google Calendar via OAuth2 |
|
|
| `flynn gdocs-auth` | Authenticate with Google Docs via OAuth2 |
|
|
| `flynn gdrive-auth` | Authenticate with Google Drive via OAuth2 |
|
|
| `flynn gtasks-auth` | Authenticate with Google Tasks via OAuth2 |
|
|
| `flynn google-auth --service <name>` | Unified Google OAuth entrypoint (`gmail`, `gcal`, `gdocs`, `gdrive`, `gtasks`) |
|
|
| `flynn gemini-auth` | Store a Gemini API key in `~/.config/flynn/auth.json` |
|
|
| `flynn skills` | List/install/manage skills |
|
|
| `flynn companion` | Run a minimal companion node client against the gateway |
|
|
|
|
`flynn setup` / `flynn onboard` now print a post-save channel verification checklist (start command, WebChat URL, `/status` smoke test, and channel-specific validation hints).
|
|
|
|
### Examples
|
|
|
|
```bash
|
|
# Start daemon with custom config
|
|
flynn start --config ~/my-config.yaml
|
|
|
|
# One-shot query
|
|
flynn send "What's the weather in London?"
|
|
|
|
# Check system health
|
|
flynn doctor --config ~/.config/flynn/config.yaml
|
|
|
|
# Treat warnings as failures (useful in CI)
|
|
flynn doctor --strict
|
|
|
|
# Show current config (secrets masked)
|
|
flynn config
|
|
|
|
# List sessions
|
|
flynn sessions
|
|
|
|
# Create backup now (uses config backup + MinIO settings)
|
|
flynn backup
|
|
|
|
# Generate shell completions
|
|
flynn completion bash # Print bash completions to stdout
|
|
flynn completion zsh --install # Install zsh completions to ~/.zshrc
|
|
flynn completion fish --install # Install fish completions to config
|
|
```
|
|
|
|
## Skills Registry
|
|
|
|
Flynn supports registry-backed skill discovery and install-by-id.
|
|
|
|
Registry source configuration:
|
|
|
|
```yaml
|
|
skills:
|
|
registry_source: ~/.config/flynn/skills-registry.json
|
|
```
|
|
|
|
You can also set `FLYNN_SKILLS_REGISTRY_SOURCE` instead of `skills.registry_source`.
|
|
|
|
Common commands:
|
|
|
|
```bash
|
|
# Discover registry entries
|
|
flynn skills registry list
|
|
flynn skills registry list --search todo --publisher acme --json
|
|
flynn skills registry show todoist
|
|
|
|
# Install by registry id
|
|
flynn skills install --registry-id todoist
|
|
|
|
# For non-local registry sources (git/archive), explicit confirmation is required
|
|
flynn skills install --registry-id todoist --confirm
|
|
```
|
|
|
|
Registry metadata (`publisher`, `homepage`, `sha256`) is treated as declared and unverified. Skill scanner checks still run before installation succeeds.
|
|
|
|
## Configuration
|
|
|
|
Config location: `~/.config/flynn/config.yaml` (or set `FLYNN_CONFIG`)
|
|
|
|
```yaml
|
|
telegram:
|
|
bot_token: "your-telegram-bot-token"
|
|
allowed_chat_ids: [123456789] # Your Telegram user ID
|
|
require_mention: false # Default false: respond in allowed groups without @mention
|
|
|
|
# Optional: Matrix
|
|
matrix:
|
|
homeserver_url: "https://matrix.example.org"
|
|
access_token: "${MATRIX_ACCESS_TOKEN}"
|
|
allowed_room_ids: ["!room1:example.org"]
|
|
require_mention: true
|
|
|
|
# Optional: Signal (signal-cli)
|
|
signal:
|
|
account: "+15551234567"
|
|
signal_cli_path: "signal-cli"
|
|
allowed_numbers: ["+15550001111"]
|
|
allowed_group_ids: []
|
|
require_mention: true
|
|
mention_name: "flynn"
|
|
poll_interval_ms: 5000
|
|
send_timeout_ms: 15000
|
|
|
|
# Optional: Microsoft Teams (Bot Framework)
|
|
teams:
|
|
app_id: "${TEAMS_APP_ID}"
|
|
app_password: "${TEAMS_APP_PASSWORD}"
|
|
allowed_conversation_ids: []
|
|
require_mention: true
|
|
|
|
# Bot Framework messaging endpoint should point to:
|
|
# POST https://<your-flynn-host>/teams/events
|
|
|
|
# Optional: Mattermost
|
|
mattermost:
|
|
server_url: "${MATTERMOST_SERVER_URL}"
|
|
bot_token: "${MATTERMOST_BOT_TOKEN}"
|
|
allowed_channel_ids: [] # Recommended: explicit channel IDs
|
|
require_mention: true
|
|
mention_name: "flynn"
|
|
poll_interval_ms: 3000
|
|
|
|
# Optional: Google Chat
|
|
google_chat:
|
|
service_account_key_file: "~/.config/flynn/google-chat-service-account.json"
|
|
# or:
|
|
# service_account_json: "${GOOGLE_CHAT_SERVICE_ACCOUNT_JSON}"
|
|
webhook_token: "${GOOGLE_CHAT_WEBHOOK_TOKEN}"
|
|
allowed_space_names: []
|
|
require_mention: true
|
|
|
|
# Google Chat messaging endpoint should point to:
|
|
# POST https://<your-flynn-host>/google-chat/events
|
|
|
|
# Optional: iMessage via BlueBubbles
|
|
bluebubbles:
|
|
endpoint: "http://localhost:1234"
|
|
api_key: "${BLUEBUBBLES_API_KEY}"
|
|
webhook_token: "${BLUEBUBBLES_WEBHOOK_TOKEN}"
|
|
allowed_chat_guids: []
|
|
require_mention: true
|
|
mention_name: "flynn"
|
|
|
|
# BlueBubbles webhook endpoint should point to:
|
|
# POST https://<your-flynn-host>/bluebubbles/events
|
|
|
|
# Optional: LINE
|
|
line:
|
|
channel_access_token: "${LINE_CHANNEL_ACCESS_TOKEN}"
|
|
channel_secret: "${LINE_CHANNEL_SECRET}"
|
|
allowed_source_ids: [] # Empty = allow all users/groups/rooms
|
|
require_mention: true
|
|
mention_name: "flynn"
|
|
# Binary attachments: when backup.minio.enabled is configured, Flynn uploads
|
|
# LINE binary attachments to MinIO and sends a share URL automatically.
|
|
|
|
# LINE webhook endpoint should point to:
|
|
# POST https://<your-flynn-host>/line/events
|
|
|
|
# Optional: Feishu / Lark
|
|
feishu:
|
|
app_id: "${FEISHU_APP_ID}"
|
|
app_secret: "${FEISHU_APP_SECRET}"
|
|
webhook_token: "${FEISHU_WEBHOOK_TOKEN}"
|
|
allowed_chat_ids: []
|
|
require_mention: true
|
|
mention_name: "flynn"
|
|
endpoint: "https://open.feishu.cn"
|
|
|
|
# Feishu webhook endpoint should point to:
|
|
# POST https://<your-flynn-host>/feishu/events
|
|
|
|
# Optional: Zalo
|
|
zalo:
|
|
oa_access_token: "${ZALO_OA_ACCESS_TOKEN}"
|
|
webhook_token: "${ZALO_WEBHOOK_TOKEN}"
|
|
allowed_user_ids: []
|
|
require_mention: true
|
|
mention_name: "flynn"
|
|
endpoint: "https://openapi.zalo.me"
|
|
# Binary attachments: when backup.minio.enabled is configured, Flynn uploads
|
|
# Zalo binary attachments to MinIO and sends a share URL automatically.
|
|
|
|
# Zalo webhook endpoint should point to:
|
|
# POST https://<your-flynn-host>/zalo/events
|
|
|
|
models:
|
|
default:
|
|
provider: anthropic
|
|
model: claude-opus-4-5-20251101
|
|
api_key: sk-ant-api03-...
|
|
# api_keys: [sk-ant-primary-..., sk-ant-secondary-...] # Optional rotation pool
|
|
# auth_profile_cooldown_ms: 30000 # Optional cooldown before retrying a failed key profile
|
|
local:
|
|
provider: ollama
|
|
model: qwen2.5:14b
|
|
fallback_chain: [local]
|
|
|
|
hooks:
|
|
confirm: [shell.*, file.write, file.patch]
|
|
log: [web.*, file.read]
|
|
silent: [notify]
|
|
```
|
|
|
|
## Safety Model
|
|
|
|
Flynn is designed to be safe-by-default when expanded beyond "chat":
|
|
|
|
- **Tool policy** restricts which tools are even available to a given context (profiles + allow/deny + per-agent/per-provider overrides).
|
|
- **Skills** can declare explicit capabilities (`manifest.json.permissions`) which are enforced at runtime.
|
|
- **Sandboxing** can isolate high-risk execution (shell/process) per-session via Docker.
|
|
- **Prompt-injection hardening** treats fetched content/tool output as untrusted data and blocks obviously unsafe tool calls when untrusted content is present.
|
|
- **Audit logs** record tool usage and approvals with redaction.
|
|
|
|
Details: `docs/security/SAFE_PERSONAL_AGENT.md`
|
|
|
|
## Agent-Oriented Architecture Diagram
|
|
|
|
If you want a fast mental model of where to start as an AI agent / contributor:
|
|
|
|
- `docs/architecture/AGENT_DIAGRAM.md`
|
|
- `docs/architecture/CONTRIBUTOR_MAP.md`
|
|
- `docs/architecture/TYPESCRIPT_MAP.md`
|
|
- `docs/architecture/SYMBOL_INDEX.md`
|
|
|
|
### Model Providers
|
|
|
|
| Provider | Config |
|
|
|----------|--------|
|
|
| Anthropic | `provider: anthropic`, `api_key`/`api_keys` or `auth_token` |
|
|
| OpenAI | `provider: openai`, `api_key`/`api_keys`, optional `endpoint` |
|
|
| Vercel AI Gateway | `provider: vercel`, `api_key` or `AI_GATEWAY_API_KEY`, optional `endpoint` |
|
|
| GitHub Copilot | `provider: github`, auto-login via OAuth device flow |
|
|
| Gemini | `provider: gemini`, `api_key` |
|
|
| Bedrock | `provider: bedrock`, AWS credentials |
|
|
| Ollama | `provider: ollama`, `model`, optional `endpoint` |
|
|
| Zhipu AI (GLM) | `provider: zhipuai`, `api_key` or `ZHIPUAI_API_KEY`, optional `endpoint` |
|
|
| xAI (Grok) | `provider: xai`, `api_key`/`api_keys` or `XAI_API_KEY` |
|
|
| MiniMax | `provider: minimax`, `api_key`/`api_keys` or `MINIMAX_API_KEY`, optional `endpoint` |
|
|
| Moonshot (Kimi) | `provider: moonshot`, `api_key`/`api_keys` or `MOONSHOT_API_KEY`, optional `endpoint` |
|
|
| llama.cpp | `provider: llamacpp`, `endpoint` |
|
|
|
|
### Model Tiers
|
|
|
|
Configure multiple models for different purposes:
|
|
|
|
```yaml
|
|
models:
|
|
fast: { provider: anthropic, model: claude-sonnet-4-... }
|
|
default: { provider: anthropic, model: claude-opus-4-5-... }
|
|
complex: { provider: anthropic, model: claude-opus-4-5-... }
|
|
local: { provider: ollama, model: qwen2.5:14b }
|
|
```
|
|
|
|
Each tier can optionally specify `auth_mode` (`auto` | `api_key` | `oauth`) to control whether Flynn uses API keys vs OAuth/token auth for that provider. `use_oauth: true` remains supported as a compatibility alias for `auth_mode: oauth`.
|
|
|
|
When multiple keys are configured via `api_keys`, Flynn rotates across key profiles on provider failures and sticks to the last successful profile until it fails. Set `auth_profile_cooldown_ms` to temporarily cool down failing profiles before retrying them.
|
|
|
|
Note: with `provider: openai` + `auth_mode: oauth` (Codex backend), Flynn currently does not send tool definitions to the provider. Tool execution is therefore unavailable in that mode, and any textual `tool_use` output should be treated as non-executable model text.
|
|
|
|
Note: with `provider: ollama`, tool execution depends on model capabilities. If Ollama reports that the selected model does not support tools, Flynn omits tool definitions for that request.
|
|
|
|
Note: with `provider: llamacpp`, tool execution depends on the served model/template correctly emitting OpenAI-style `tool_calls`. Models/templates that do not preserve tool-call structure may fall back to plain text behavior.
|
|
|
|
### Agent Backends
|
|
|
|
Flynn can run with the built-in native backend or delegate message processing to external CLI backends.
|
|
|
|
```yaml
|
|
backends:
|
|
default: codex
|
|
native: { enabled: true }
|
|
codex: { enabled: false, path: /usr/local/bin/codex, args: [], timeout_ms: 120000 }
|
|
claude_code: { enabled: false, path: /usr/local/bin/claude, args: [], timeout_ms: 120000 }
|
|
opencode: { enabled: false, path: /usr/local/bin/opencode, args: [], timeout_ms: 120000 }
|
|
gemini: { enabled: false, path: /usr/local/bin/gemini, args: [], timeout_ms: 120000 }
|
|
pi_embedded:
|
|
enabled: false
|
|
timeout_ms: 120000
|
|
no_tools_mode: true
|
|
model: openclaw-default
|
|
system_prompt_mode: hybrid # flynn | pi_default | hybrid
|
|
module: "@mariozechner/pi-agent-core" # optional module override
|
|
```
|
|
|
|
`pi_embedded` is intended for canary migration cohorts. In spike mode (`no_tools_mode: true`), Flynn keeps tool-oriented turns on native and only routes plain-text turns to Pi.
|
|
|
|
To evaluate canary performance from audit logs, run:
|
|
|
|
```bash
|
|
pnpm audit:backend-canary \
|
|
--audit ~/.local/share/flynn/audit.log \
|
|
--backend pi_embedded \
|
|
--baseline native \
|
|
--session telegram:8367012007 \
|
|
--gate-min-target-routes 8 \
|
|
--gate-min-baseline-routes 2 \
|
|
--gate-min-target-attempts 8 \
|
|
--format markdown
|
|
```
|
|
|
|
Phase-2 evaluation checklist and decision template: `docs/plans/pi_embedded_evaluation.md`.
|
|
|
|
When `args` is non-empty:
|
|
- use `{prompt}` in an argument to inject the full generated prompt directly into argv.
|
|
- if `{prompt}` is not present, Flynn appends backend-specific prompt args.
|
|
|
|
If multiple external backends are enabled, set `backends.default` to choose explicitly. If omitted, Flynn selects by priority: `codex` -> `claude_code` -> `opencode` -> `gemini` -> `pi_embedded`.
|
|
|
|
You can also route specific named agents to a backend:
|
|
|
|
```yaml
|
|
agent_configs:
|
|
coder:
|
|
model_tier: complex
|
|
backend: codex # native | codex | claude_code | opencode | gemini | pi_embedded
|
|
pi_canary:
|
|
model_tier: default
|
|
backend: pi_embedded
|
|
```
|
|
|
|
### Native Audio Support
|
|
|
|
Voice messages from channels can be handled in two ways:
|
|
|
|
1. **Native passthrough** -- Audio sent directly to models that support audio input (Gemini, OpenAI, GitHub). No transcription step needed.
|
|
2. **Whisper transcription** -- Audio transcribed to text via a Whisper-compatible API, then sent as text to models that don't support audio input (Anthropic, Bedrock, Ollama, llama.cpp).
|
|
|
|
Flynn automatically routes based on the model's capabilities. You can override this per-tier:
|
|
|
|
```yaml
|
|
models:
|
|
default:
|
|
provider: gemini
|
|
model: gemini-2.0-flash
|
|
supports_audio: true # Force native audio (auto-detected for known providers)
|
|
fast:
|
|
provider: anthropic
|
|
model: claude-sonnet-4
|
|
supports_audio: false # Force transcription (default for Anthropic)
|
|
```
|
|
|
|
### Audio Transcription
|
|
|
|
Configure a Whisper-compatible endpoint for models that don't support native audio:
|
|
|
|
```yaml
|
|
audio:
|
|
enabled: true
|
|
provider:
|
|
type: custom # openai, groq, ollama, llamacpp, custom
|
|
endpoint: "http://localhost:18801/v1/audio/transcriptions"
|
|
api_key: "${WHISPER_API_KEY}" # Optional Bearer token
|
|
model: "whisper-1" # Model name (default: whisper-1)
|
|
```
|
|
|
|
| Field | Required | Description |
|
|
|-------|----------|-------------|
|
|
| `enabled` | no | Enable audio transcription (default: `false`) |
|
|
| `provider.type` | yes | Provider type: `openai`, `groq`, `ollama`, `llamacpp`, or `custom` |
|
|
| `provider.endpoint` | yes | Whisper-compatible API endpoint |
|
|
| `provider.api_key` | no | Bearer token for authentication |
|
|
| `provider.model` | no | Model name sent in request (default: `whisper-1`) |
|
|
| `talk_mode.enabled` | no | Enable wake-phrase talk mode gating (default: `false`) |
|
|
| `talk_mode.wake_phrase` | no | Phrase that activates talk mode (default: `hey flynn`) |
|
|
| `talk_mode.timeout_ms` | no | Active listen window after wake (default: `120000`) |
|
|
| `talk_mode.allow_manual_toggle` | no | Enable `/talk on|off|status` controls (default: `true`) |
|
|
|
|
Without an `audio` config, voice messages from non-audio-capable models will display an error message to the user. For local transcription, you can run a whisper.cpp server:
|
|
|
|
```bash
|
|
# Option 1: Manual docker run
|
|
docker run -d \
|
|
--name whisper-server \
|
|
-p 18801:8080 \
|
|
ghcr.io/ggml-org/whisper.cpp:main \
|
|
--model /app/models/ggml-base.en.bin \
|
|
--host 0.0.0.0 \
|
|
--port 8080 \
|
|
--convert \
|
|
--language en \
|
|
--inference-path /v1/audio/transcriptions
|
|
|
|
# Option 2: Using docker-compose profile (service is pre-defined in docker-compose.yml)
|
|
docker compose --profile voice up -d whisper-server
|
|
```
|
|
|
|
Audio persistence and diagnostics:
|
|
|
|
- Latest inbound voice bytes are stored per-session in `~/.local/share/flynn/sessions.db` under `session_config.key = "lastAudioAttachment"` (used to safely hydrate `audio.transcribe` calls).
|
|
- `/reset` clears session history and session config for that chat, including `lastAudioAttachment`.
|
|
- To clear one session manually: `DELETE FROM session_config WHERE session_id='<channel:sender>' AND key='lastAudioAttachment'`.
|
|
- When Flynn rewrites bad model-provided audio tool args, it emits audit event `tool.args_rewritten`.
|
|
- Runbook: `docs/runbooks/VOICE_TRANSCRIPTION_DEBUG.md`.
|
|
- SQLite quick reference: `docs/runbooks/SQLITE_QUICK_REFERENCE.md`.
|
|
|
|
### Web Search
|
|
|
|
Flynn provides web search tools through the `web_search` config:
|
|
|
|
- `web.search` - general web results (Brave or SearXNG provider)
|
|
- `web.search.news` - news results (Brave provider only)
|
|
|
|
Brave Search configuration:
|
|
|
|
```yaml
|
|
web_search:
|
|
provider: brave
|
|
api_key: "${BRAVE_API_KEY}"
|
|
max_results: 5
|
|
```
|
|
|
|
SearXNG configuration:
|
|
|
|
```yaml
|
|
web_search:
|
|
provider: searxng
|
|
endpoint: "http://searxng:8080"
|
|
fallback_endpoint: "https://searxng.homelab.local" # Optional backup endpoint
|
|
max_results: 5
|
|
```
|
|
|
|
| Field | Required | Description |
|
|
|-------|----------|-------------|
|
|
| `provider` | no | `brave` or `searxng` (default: `brave`) |
|
|
| `api_key` | yes (Brave) | Brave Search API key |
|
|
| `endpoint` | yes (SearXNG primary) | Primary base URL for self-hosted SearXNG |
|
|
| `fallback_endpoint` | no (SearXNG) | Backup SearXNG base URL used if primary endpoint fails |
|
|
| `max_results` | no | Default result count for `web.search*` tools (default: `5`, max: `20`) |
|
|
|
|
Brave-specific query fields (supported by `web.search`, and by `web.search.news` except `safeSearch`):
|
|
|
|
- `country` (for example `us`, `gb`)
|
|
- `searchLang` (for example `en`, `de`)
|
|
- `freshness` (`pd`, `pw`, `pm`, `py`)
|
|
- `safeSearch` (`off`, `moderate`, `strict`) - `web.search` only
|
|
|
|
Optional local Brave Search container (for MCP-based workflows):
|
|
|
|
```bash
|
|
docker compose --profile search up -d brave-search
|
|
```
|
|
|
|
Optional local SearXNG container (for self-hosted web search):
|
|
|
|
```bash
|
|
docker compose --profile search up -d searxng
|
|
```
|
|
|
|
Local-first + homelab fallback pattern:
|
|
|
|
- If Flynn runs in Docker Compose, set `endpoint: "http://searxng:8080"`.
|
|
- If Flynn runs on the host (`pnpm dev` / `pnpm start`), set `endpoint: "http://localhost:18803"`.
|
|
- Set `fallback_endpoint` to your existing homelab SearXNG URL.
|
|
|
|
Example for host-run Flynn with local compose SearXNG and homelab backup:
|
|
|
|
```yaml
|
|
web_search:
|
|
provider: searxng
|
|
endpoint: "http://localhost:18803"
|
|
fallback_endpoint: "https://searxng.homelab.local"
|
|
max_results: 5
|
|
```
|
|
|
|
Note: Flynn's built-in `web.search*` tools call Brave's HTTP API directly and use `web_search.api_key`; they do not route through the `brave-search` MCP container.
|
|
|
|
### Text-to-Speech (TTS) Reply Audio
|
|
|
|
Flynn can attach synthesized voice replies (OpenAI-compatible `/v1/audio/speech`) alongside text responses.
|
|
|
|
```yaml
|
|
tts:
|
|
enabled: true
|
|
enabled_channels: [telegram, whatsapp, discord] # Empty = all channels
|
|
provider:
|
|
type: openai # openai | custom
|
|
endpoint: "https://api.openai.com/v1/audio/speech"
|
|
api_key: "${OPENAI_API_KEY}" # Optional Bearer token
|
|
model: "gpt-4o-mini-tts"
|
|
voice: "alloy"
|
|
format: "mp3" # mp3 | wav | opus
|
|
```
|
|
|
|
| Field | Required | Description |
|
|
|-------|----------|-------------|
|
|
| `tts.enabled` | no | Enable voice reply synthesis (default: `false`) |
|
|
| `tts.enabled_channels` | no | Channels allowed to receive voice replies (`[]` means all channels) |
|
|
| `tts.provider.type` | no | `openai` or `custom` (default: `openai`) |
|
|
| `tts.provider.endpoint` | no | OpenAI-compatible `/v1/audio/speech` endpoint (`openai` defaults to OpenAI API URL) |
|
|
| `tts.provider.api_key` | no | Bearer token for authentication |
|
|
| `tts.provider.model` | no | TTS model (default: `gpt-4o-mini-tts`) |
|
|
| `tts.provider.voice` | no | Voice identifier (default: `alloy`) |
|
|
| `tts.provider.format` | no | Output format: `mp3`, `wav`, `opus` (default: `mp3`) |
|
|
|
|
### Capture Tools
|
|
|
|
Flynn includes host capture tools:
|
|
- `screen.capture` -> captures current screen and returns base64 image payload
|
|
- `camera.capture` -> captures one camera frame and returns base64 image payload
|
|
|
|
Notes:
|
|
- These are host-command wrappers and require platform binaries:
|
|
- macOS: `screencapture` (screen), `imagesnap` (camera)
|
|
- Linux: `grim` or ImageMagick `import` (screen), `ffmpeg` (camera)
|
|
|
|
## Telegram Commands
|
|
|
|
| Command | Description |
|
|
|---------|-------------|
|
|
| `/start` | Initialize bot |
|
|
| `/reset` | Clear conversation history |
|
|
| `/status` | Show current model and status |
|
|
| `/local` | Switch to local model |
|
|
| `/cloud` | Switch to cloud model |
|
|
| `/model` | Show model info and options |
|
|
| `/approvals` | List pending approval gates for current session |
|
|
| `/approve [id]` | Approve latest (or specific) pending gate |
|
|
| `/deny [id] [reason]` | Deny latest (or specific) pending gate |
|
|
| `/skill <list|search|install>` | In-chat skill discovery/install (`list`, `search <term>`, `install <registry-id>`) |
|
|
|
|
## Web UI Dashboard
|
|
|
|
Flynn includes a built-in web control dashboard served by the WebSocket gateway. Access it at `http://localhost:18800` (or your configured gateway port).
|
|
|
|
### Pages
|
|
|
|
| Page | Description |
|
|
|------|-------------|
|
|
| **Dashboard** | System health cards, channel status, usage stats, assistant-health quick actions, playbook presets with rollback, activation checklist, and daily briefing preview/test-send. Auto-refreshes every 10s |
|
|
| **Chat** | Session selector, streaming tool events, markdown rendering with syntax highlighting |
|
|
| **Sessions** | Browse all sessions, view message history, delete sessions |
|
|
| **Usage** | Token usage summary cards, per-session breakdown table, auto-refresh |
|
|
| **Settings** | Edit hook patterns (confirm/log/silent), Personal Assistant Mode toggles, view tools/services, and redacted config |
|
|
|
|
The dashboard is a vanilla JS SPA with no build step — hash-based routing, ES modules, and the existing WebSocket JSON-RPC protocol.
|
|
|
|
## Terminal UI
|
|
|
|
```bash
|
|
# Minimal mode (readline)
|
|
pnpm tui
|
|
|
|
# Fullscreen mode (React/Ink)
|
|
pnpm tui:fs
|
|
```
|
|
|
|
### TUI Commands
|
|
|
|
| Command | Description |
|
|
|---------|-------------|
|
|
| `/help` | Show help |
|
|
| `/model` | Show all model tiers and which is active |
|
|
| `/model <tier>` | Switch active tier (`local`, `default`, `fast`, `complex`, or aliases `ollama`, `sonnet`, `haiku`, `opus`) |
|
|
| `/model <tier> <provider/model>` | Hot-swap a tier's provider and model at runtime |
|
|
| `/backend [provider]` | Show or switch local backend (`ollama`, `llamacpp`) |
|
|
| `/login [provider]` | Authenticate with GitHub (OAuth device flow) |
|
|
| `/reset` | Clear history |
|
|
| `/status` | Show session info |
|
|
| `/tools` | Show authoritative runtime tool list for this session |
|
|
| `/research <task>` | Delegate a task to `agent_configs.research` |
|
|
| `/council <task>` | Run dual D/P councils pipeline with bridge+meta merge (brief in TUI, full artifacts saved to disk) |
|
|
| `/compact` | Compact conversation context |
|
|
| `/usage` | Show token usage and cost |
|
|
| `/context` | Show estimated context-window usage |
|
|
| `/verbose` | Toggle verbose output mode |
|
|
| `/pair` | Generate/list/revoke DM pairing codes |
|
|
| `/fullscreen` | Switch to fullscreen mode |
|
|
| `/transfer <dest>` | Transfer session to another frontend (`telegram` or `tui`) |
|
|
| `/approvals` | List pending approval gates for current session |
|
|
| `/approve [id]` | Approve latest (or specific) pending gate |
|
|
| `/deny [id] [reason]` | Deny latest (or specific) pending gate |
|
|
| `/skill <list|search|install>` | In-chat skill discovery/install (`list`, `search <term>`, `install <registry-id>`) |
|
|
| `/quit` | Exit |
|
|
|
|
TUI keyboard controls: `Esc` cancels active prompt/running response. `Ctrl+C` clears the current input; press `Ctrl+C` twice quickly to exit.
|
|
|
|
#### Runtime Model Switching
|
|
|
|
Switch providers and models on the fly without editing config or restarting:
|
|
|
|
```bash
|
|
# Show current tiers
|
|
/model
|
|
|
|
# Switch active tier
|
|
/model fast
|
|
/model complex
|
|
|
|
# Hot-swap a tier's provider/model
|
|
/model default anthropic/claude-sonnet-4
|
|
/model default zhipuai/glm-4.7
|
|
/model fast github/gpt-4o-mini
|
|
/model local ollama/glm-4.7-flash
|
|
```
|
|
|
|
The provider name must match a supported provider (`anthropic`, `openai`, `gemini`, `ollama`, `llamacpp`, `openrouter`, `vercel`, `bedrock`, `github`, `zhipuai`, `xai`, `minimax`, `moonshot`, `synthetic`). Tab completion is available for both tiers and provider names.
|
|
|
|
For cloud Zhipu models, ensure `ZHIPUAI_API_KEY` is set or `api_key` is configured in the relevant tier.
|
|
|
|
**Note:** The `/model` command works in the TUI only. WebChat sessions inherit the active tier from the daemon.
|
|
|
|
### Research Agent Quickstart
|
|
|
|
Add a dedicated research sub-agent and delegate to it with `/research`:
|
|
|
|
```yaml
|
|
agent_configs:
|
|
research:
|
|
model_tier: complex
|
|
tool_profile: messaging
|
|
system_prompt: |
|
|
You are a research agent. Find, verify, and synthesize information.
|
|
Prefer primary sources. Include links and concrete dates.
|
|
Keep output structured and concise.
|
|
```
|
|
|
|
Then use:
|
|
|
|
```bash
|
|
/research compare k0s vs k3s for a 3-node homelab in 2026
|
|
```
|
|
|
|
You can also trigger this without the slash command:
|
|
|
|
```text
|
|
research compare k0s vs k3s for a 3-node homelab in 2026
|
|
look up best practices for k8s backup retention
|
|
```
|
|
|
|
If the `research` agent is not configured, Flynn returns a setup hint with available agent names.
|
|
`flynn setup` can configure this under the Security section.
|
|
|
|
### Councils Pipeline Quickstart
|
|
|
|
Enable deterministic dual-council orchestration and define required role agents:
|
|
|
|
```yaml
|
|
councils:
|
|
enabled: true
|
|
|
|
agent_configs:
|
|
council_d_arbiter:
|
|
model_tier: default
|
|
tool_profile: messaging
|
|
system_prompt: You are the D arbiter. Score and shortlist ideas conservatively.
|
|
council_d_freethinker:
|
|
model_tier: default
|
|
tool_profile: messaging
|
|
system_prompt: You are the D freethinker. Generate and ground ideas in JSON only.
|
|
council_p_arbiter:
|
|
model_tier: default
|
|
tool_profile: messaging
|
|
system_prompt: You are the P arbiter. Prefer novelty and leverage while scoring.
|
|
council_p_freethinker:
|
|
model_tier: default
|
|
tool_profile: messaging
|
|
system_prompt: You are the P freethinker. Generate contrarian ideas in JSON only.
|
|
council_meta_arbiter:
|
|
model_tier: default
|
|
tool_profile: messaging
|
|
system_prompt: You are the meta arbiter. Select from provided IDs only; no novel mechanisms.
|
|
```
|
|
|
|
Run from chat:
|
|
|
|
```bash
|
|
/council design a 30-day plan to cut CI flakiness by 50%
|
|
```
|
|
|
|
By default, `/council` prints a concise execution brief in the TUI and persists full artifacts under:
|
|
|
|
```text
|
|
~/.local/share/flynn/councils/
|
|
```
|
|
|
|
When `FLYNN_DATA_DIR` is set, artifacts are written to:
|
|
|
|
```text
|
|
${FLYNN_DATA_DIR}/councils/
|
|
```
|
|
|
|
Each run saves:
|
|
- `*.md` summary report (brief + full conversation trace + raw JSON block)
|
|
- `*.json` full structured `CouncilRunResult`
|
|
|
|
Or via tools:
|
|
|
|
```json
|
|
{"name":"council.run","args":{"task":"design a 30-day plan to cut CI flakiness by 50%"}}
|
|
```
|
|
|
|
## Running as Service
|
|
|
|
```bash
|
|
# Create systemd user service
|
|
mkdir -p ~/.config/systemd/user
|
|
|
|
cat > ~/.config/systemd/user/flynn.service << 'EOF'
|
|
[Unit]
|
|
Description=Flynn Personal AI Assistant
|
|
After=network.target ollama.service
|
|
|
|
[Service]
|
|
Type=simple
|
|
WorkingDirectory=/path/to/flynn
|
|
ExecStart=/usr/bin/pnpm start
|
|
Restart=on-failure
|
|
RestartSec=5
|
|
Environment=NODE_ENV=production
|
|
|
|
[Install]
|
|
WantedBy=default.target
|
|
EOF
|
|
|
|
# Enable and start
|
|
systemctl --user daemon-reload
|
|
systemctl --user enable --now flynn
|
|
|
|
# View logs
|
|
journalctl --user -u flynn -f
|
|
```
|
|
|
|
## Hook Engine
|
|
|
|
Control sensitive operations with pattern matching:
|
|
|
|
```yaml
|
|
hooks:
|
|
confirm: # Requires user approval via Telegram
|
|
- shell.*
|
|
- process.start
|
|
- process.kill
|
|
- browser.*
|
|
- message.send
|
|
- cron.create
|
|
- cron.delete
|
|
- file.write
|
|
- file.patch
|
|
log: # Logs but doesn't block
|
|
- web.*
|
|
- file.read
|
|
silent: # Executes without notification
|
|
- notify
|
|
```
|
|
|
|
For unrestricted deployments, pair hooks with agent-level sensitive gating:
|
|
|
|
```yaml
|
|
agents:
|
|
# deny_without_elevation | confirm_without_elevation
|
|
sensitive_mode: confirm_without_elevation
|
|
immutable_denylist:
|
|
- tool: shell.exec
|
|
args_pattern: "git push origin main"
|
|
reason: "direct main pushes are blocked"
|
|
- tool: shell.exec
|
|
args_pattern: "git reset --hard"
|
|
reason: "destructive hard reset is blocked"
|
|
```
|
|
|
|
## Browser Automation Tools
|
|
|
|
Flynn ships these browser tools:
|
|
|
|
- `browser.navigate`
|
|
- `browser.screenshot`
|
|
- `browser.click`
|
|
- `browser.type`
|
|
- `browser.content`
|
|
- `browser.eval`
|
|
- `browser.evaluate` (alias of `browser.eval`)
|
|
|
|
These tools are backed by a Puppeteer/CDP browser manager and are only registered when `browser.enabled: true`.
|
|
They can still be filtered out by tool policy (`tools.profile`, `tools.allow`, `tools.deny`).
|
|
At startup, Flynn logs the browser tools that remain available after policy filtering.
|
|
|
|
```yaml
|
|
browser:
|
|
enabled: true
|
|
headless: true
|
|
max_pages: 5
|
|
default_timeout: 30000
|
|
# executable_path: /usr/bin/google-chrome
|
|
# ws_endpoint: ws://127.0.0.1:9222/devtools/browser/<id>
|
|
|
|
tools:
|
|
profile: coding # or full
|
|
```
|
|
|
|
## Cron Scheduling
|
|
|
|
Schedule automated messages on cron schedules. Each job fires an inbound message through the agent pipeline and routes the response to a configured output channel.
|
|
|
|
Set `automation.delivery_mode` to control automation session behavior:
|
|
- `shared_session` (default): reuse one session per cron job/webhook name.
|
|
- `isolated_job`: create a fresh session per cron trigger/webhook request.
|
|
- `announce`: create a fresh ephemeral announce-style run per trigger (no shared automation history).
|
|
|
|
```yaml
|
|
automation:
|
|
delivery_mode: shared_session
|
|
reactions:
|
|
- name: boss-email
|
|
on: [gmail]
|
|
filter:
|
|
contains: "boss@company.com"
|
|
run: |
|
|
Summarize this email and propose next actions.
|
|
|
|
{{text}}
|
|
cron:
|
|
- name: daily-summary
|
|
schedule: "0 9 * * *" # 9 AM daily
|
|
message: "Give me a summary of today's tasks"
|
|
output:
|
|
channel: telegram # Route response to Telegram
|
|
peer: "123456789" # Chat ID to send to
|
|
timezone: Europe/London # Optional timezone
|
|
enabled: true
|
|
once_per_local_day: false # Optional dedupe: max one trigger/day in job timezone
|
|
|
|
- name: hourly-check
|
|
schedule: "0 * * * *" # Every hour
|
|
message: "Check system status"
|
|
output:
|
|
channel: telegram
|
|
peer: "123456789"
|
|
enabled: false # Disabled, won't fire
|
|
model_tier: fast # Use fast tier for quick checks
|
|
|
|
# Optional built-in daily briefing preset.
|
|
# This automatically registers a cron job; you only set schedule/output/prompt.
|
|
daily_briefing:
|
|
enabled: true
|
|
name: daily-briefing
|
|
schedule: "0 8 * * *"
|
|
timezone: America/New_York
|
|
dedupe_per_local_day: true
|
|
output:
|
|
channel: telegram
|
|
peer: "123456789"
|
|
model_tier: fast
|
|
prompt: |
|
|
Create my daily briefing.
|
|
Summarize today's calendar, unread/important email, and top pending tasks.
|
|
|
|
# Optional scheduled MinIO -> memory synchronization.
|
|
# Runs direct ingestion jobs without requiring interactive tool calls.
|
|
minio_sync:
|
|
enabled: true
|
|
interval: "6h"
|
|
run_on_start: false
|
|
notify_on_success: false
|
|
tasks:
|
|
- prefix: "knowledge/"
|
|
namespace_base: "global/knowledge/minio"
|
|
mode: append
|
|
max_objects: 20
|
|
max_chars_per_object: 8000
|
|
force: false
|
|
```
|
|
|
|
### Cron Config Fields
|
|
|
|
| Field | Required | Description |
|
|
|-------|----------|-------------|
|
|
| `automation.delivery_mode` | no | Automation session strategy: `shared_session`, `isolated_job`, or `announce` (default: `shared_session`) |
|
|
| `automation.reactions.*` | no | Event-triggered prompt rewrite rules for inbound automation/chat events (supports channel matching + text/metadata filters) |
|
|
| `name` | yes | Unique job identifier |
|
|
| `schedule` | yes | Cron expression (standard 5-field) |
|
|
| `message` | yes | Text sent to the agent when the job fires |
|
|
| `output.channel` | yes | Channel name to route the response (e.g. `telegram`) |
|
|
| `output.peer` | yes | Peer/chat ID on the output channel |
|
|
| `timezone` | no | IANA timezone (defaults to system timezone) |
|
|
| `enabled` | no | Whether the job is active (default: `true`) |
|
|
| `model_tier` | no | Model tier for this job: `fast`, `default`, `complex`, or `local` |
|
|
| `once_per_local_day` | no | If true, suppress duplicate triggers within the same local day (job timezone) |
|
|
| `automation.daily_briefing.*` | no | Built-in daily briefing preset; generates an extra cron job when `enabled: true` and `output` is set |
|
|
| `automation.minio_sync.*` | no | Scheduled MinIO prefix ingestion into memory namespaces (direct daemon automation) |
|
|
|
|
### Reactions (Event -> Action Prompting)
|
|
|
|
Reactions let you convert inbound events into deterministic agent prompts without adding new webhook endpoints or cron jobs.
|
|
|
|
- Matchers support:
|
|
- channel list (`on`)
|
|
- text filters (`contains`, `regex`)
|
|
- metadata path filters (`metadata` with dot paths)
|
|
- Templates support:
|
|
- `{{text}}`, `{{channel}}`, `{{sender_id}}`
|
|
- `{{metadata.some.path}}`
|
|
- First matching rule wins.
|
|
|
|
## Backup Scheduling
|
|
|
|
Daemon backups can run on a fixed interval (`backup.interval`) or a cron schedule (`backup.schedule`). If both are set, `backup.schedule` takes precedence.
|
|
|
|
```yaml
|
|
backup:
|
|
enabled: true
|
|
schedule: "0 2 * * *" # Optional cron schedule (nightly 2 AM)
|
|
interval: "24h" # Fallback when schedule is not set
|
|
run_on_start: true # Also run once on daemon start
|
|
notify:
|
|
channel: telegram
|
|
peer: "123456789"
|
|
failure_threshold: 1 # Notify after this many consecutive failures
|
|
notify_recovery: true # Send a recovery message after failure clears
|
|
local_dir: ~/.local/share/flynn/backups
|
|
include_vectors: true
|
|
minio:
|
|
enabled: true
|
|
endpoint: localhost:9000
|
|
access_key: "${MINIO_ACCESS_KEY}"
|
|
secret_key: "${MINIO_SECRET_KEY}"
|
|
bucket: flynn-backups
|
|
prefix: flynn
|
|
secure: true
|
|
# Optional absolute path to MinIO client binary if not on PATH
|
|
mc_path: /usr/local/bin/mc
|
|
```
|
|
|
|
## MinIO Share Tool
|
|
|
|
When `backup.minio.enabled` is configured, Flynn also exposes MinIO tools:
|
|
|
|
- `minio.share`: upload a local file to the configured MinIO bucket and return a temporary download URL (`mc share download`)
|
|
- `minio.ingest`: read a text-like object (plus PDF/DOCX via extractor tools when available) from MinIO and append/replace a memory namespace
|
|
- `minio.sync`: recursively ingest a MinIO prefix into nested memory namespaces with object and size limits (including PDF/DOCX extraction when available)
|
|
|
|
PDF/DOCX ingestion runtime requirements:
|
|
|
|
- PDF extraction requires `pdftotext`.
|
|
- DOCX extraction requires `pandoc` or `docx2txt`.
|
|
- `flynn setup` now checks these dependencies after config save when `backup.minio.enabled: true`, and prints OS-aware install hints when missing.
|
|
- `flynn doctor` reports `MinIO ingest extractors` status (with install hints) so missing binaries are visible in health checks.
|
|
|
|
## Kubernetes Tools
|
|
|
|
Optional Kubernetes tools are available when `k8s.enabled: true`:
|
|
|
|
- `k8s.pods`: list pod status (phase, ready count, restarts)
|
|
- `k8s.deployments`: list deployment replica readiness
|
|
- `k8s.logs`: fetch recent pod logs
|
|
|
|
```yaml
|
|
k8s:
|
|
enabled: true
|
|
kubectl_path: kubectl
|
|
default_namespace: observability
|
|
allowed_namespaces: [observability, platform]
|
|
```
|
|
|
|
## Inbound Webhooks
|
|
|
|
HTTP endpoints that trigger agent processing. Each webhook accepts POST requests, optionally verifies an HMAC signature, renders a message template, and routes the agent's response to an output channel.
|
|
|
|
```yaml
|
|
automation:
|
|
delivery_mode: shared_session
|
|
webhooks:
|
|
- name: github-push
|
|
secret: "whsec_..." # HMAC secret for signature verification
|
|
message: "GitHub push to {{json.repository.full_name}}: {{json.head_commit.message}}"
|
|
output:
|
|
channel: telegram
|
|
peer: "123456789"
|
|
|
|
- name: alertmanager
|
|
message: "Alert: {{json.alerts.0.annotations.summary}}"
|
|
output:
|
|
channel: discord
|
|
peer: "channel-id"
|
|
```
|
|
|
|
Webhooks are available at `POST /webhooks/:name` on the gateway HTTP server. They bypass gateway token auth and use their own per-webhook HMAC verification instead.
|
|
|
|
### Webhook Config Fields
|
|
|
|
| Field | Required | Description |
|
|
|-------|----------|-------------|
|
|
| `automation.delivery_mode` | no | Automation session strategy: `shared_session`, `isolated_job`, or `announce` (default: `shared_session`) |
|
|
| `name` | yes | Unique webhook identifier (used in URL path) |
|
|
| `secret` | no | HMAC secret for `X-Webhook-Signature` header verification (SHA-256) |
|
|
| `message` | no | Template for the message sent to the agent (default: `{{body}}`) |
|
|
| `output.channel` | yes | Channel name to route the response (e.g. `telegram`) |
|
|
| `output.peer` | yes | Peer/chat ID on the output channel |
|
|
| `enabled` | no | Whether the webhook is active (default: `true`) |
|
|
|
|
### Template Variables
|
|
|
|
| Variable | Description |
|
|
|----------|-------------|
|
|
| `{{body}}` | Raw request body as string |
|
|
| `{{json.field}}` | Extract a field from JSON body (dot notation for nested fields) |
|
|
|
|
## Heartbeat Monitor
|
|
|
|
Periodic health checks that validate system components and notify a configured channel on failure.
|
|
|
|
```yaml
|
|
automation:
|
|
heartbeat:
|
|
enabled: true
|
|
interval: "5m" # Check every 5 minutes
|
|
notify_cooldown: "30m" # Suppress repeated alerts inside cooldown window
|
|
checks: [gateway, model, channels, memory, disk, process_memory, backup, provider_errors]
|
|
notify:
|
|
channel: telegram
|
|
peer: "123456789"
|
|
failure_threshold: 2 # Notify after 2 consecutive failures
|
|
disk_threshold_mb: 100 # Warn when <100MB free
|
|
process_memory_threshold_mb: 1500 # Warn when RSS memory exceeds threshold
|
|
backup_failure_threshold: 1 # Warn when backup failures meet threshold
|
|
provider_error_rate_threshold: 0.5 # Warn when provider error rate >= threshold
|
|
provider_error_min_calls: 5 # Minimum model calls per provider before evaluation
|
|
```
|
|
|
|
### Heartbeat Checks
|
|
|
|
| Check | What it validates |
|
|
|-------|-------------------|
|
|
| `gateway` | HTTP server is responsive |
|
|
| `model` | Model router has an active tier configured |
|
|
| `channels` | At least one channel adapter is connected |
|
|
| `memory` | Memory directory is readable and writable |
|
|
| `disk` | Free disk space exceeds threshold |
|
|
| `process_memory` | Flynn process RSS memory usage stays under threshold |
|
|
| `backup` | Backup scheduler consecutive failures stay under threshold |
|
|
| `provider_errors` | Model provider error rates stay below threshold |
|
|
|
|
The monitor sends a notification when failures reach the configured threshold and a recovery notification when all checks pass again.
|
|
Repeated failure/recovery notifications are throttled by `notify_cooldown`.
|
|
|
|
### Heartbeat Config Fields
|
|
|
|
| Field | Required | Description |
|
|
|-------|----------|-------------|
|
|
| `enabled` | no | Enable the heartbeat monitor (default: `false`) |
|
|
| `interval` | no | Check interval: `60s`, `5m`, `1h` (default: `5m`) |
|
|
| `notify_cooldown` | no | Minimum time between repeated heartbeat notifications of the same type (default: `30m`) |
|
|
| `checks` | no | Which checks to run (default: `gateway, model, channels, memory, disk, process_memory, backup, provider_errors`) |
|
|
| `notify.channel` | no | Channel to send failure/recovery notifications |
|
|
| `notify.peer` | no | Peer/chat ID for notifications |
|
|
| `failure_threshold` | no | Consecutive failures before notifying (default: `2`) |
|
|
| `disk_threshold_mb` | no | Disk space warning threshold in MB (default: `100`) |
|
|
| `process_memory_threshold_mb` | no | RSS memory threshold in MB for `process_memory` check (default: `1500`) |
|
|
| `backup_failure_threshold` | no | Consecutive backup failures threshold for `backup` check (default: `1`) |
|
|
| `provider_error_rate_threshold` | no | Error-rate threshold (0..1) for `provider_errors` check (default: `0.5`) |
|
|
| `provider_error_min_calls` | no | Minimum provider calls before applying error-rate threshold (default: `5`) |
|
|
|
|
### Common Schedules and Routing
|
|
|
|
- Nightly backups to Telegram alerts:
|
|
- `backup.schedule: "0 2 * * *"`
|
|
- `backup.notify.channel: telegram`
|
|
- Weekday daily briefing to Discord:
|
|
- `automation.daily_briefing.schedule: "0 8 * * 1-5"`
|
|
- `automation.daily_briefing.output.channel: discord`
|
|
- High-frequency heartbeat to Slack:
|
|
- `automation.heartbeat.interval: "2m"`
|
|
- `automation.heartbeat.notify.channel: slack`
|
|
- MinIO sync every 6h to WebChat:
|
|
- `automation.minio_sync.interval: "6h"`
|
|
- `automation.minio_sync.notify.channel: webchat`
|
|
|
|
`flynn setup` now includes an Operator Pack option in Automation that preconfigures scheduled backups, heartbeat alerts, a daily briefing, and a default MinIO sync task, with prompts for output channel/peer routing.
|
|
See `docs/operations/OPERATOR_PACK.md` for an operations runbook and verification checklist.
|
|
|
|
Example Operator Pack output routing:
|
|
|
|
```yaml
|
|
backup:
|
|
enabled: true
|
|
schedule: "0 2 * * *"
|
|
notify:
|
|
channel: telegram
|
|
peer: "123456789"
|
|
|
|
automation:
|
|
heartbeat:
|
|
enabled: true
|
|
notify:
|
|
channel: telegram
|
|
peer: "123456789"
|
|
notify_cooldown: "30m"
|
|
daily_briefing:
|
|
enabled: true
|
|
output:
|
|
channel: telegram
|
|
peer: "123456789"
|
|
minio_sync:
|
|
enabled: true
|
|
notify:
|
|
channel: telegram
|
|
peer: "123456789"
|
|
```
|
|
|
|
## Gmail Pub/Sub Watcher
|
|
|
|
Monitor a Gmail inbox and forward new messages into the agent pipeline.
|
|
|
|
Supported delivery modes:
|
|
- **Push** (Gmail watch → Pub/Sub topic → HTTP push subscription → `POST /gmail/push`)
|
|
- **Pull** (Pub/Sub pull subscription → Flynn periodically pulls messages; no inbound webhook)
|
|
- **Polling** (Gmail History API polling fallback)
|
|
|
|
### Prerequisites
|
|
|
|
1. Create a Google Cloud project with the Gmail API enabled
|
|
2. Create OAuth2 credentials (Desktop application type) and download the JSON file
|
|
3. Run `flynn gmail-auth` to complete the OAuth2 flow and store the refresh token
|
|
- Requests Gmail scopes for settings + read access (`gmail.settings.basic` + `gmail.readonly`)
|
|
- Flynn stores service tokens in `~/.config/flynn/auth.json` and keeps per-service token files for compatibility
|
|
|
|
For Pub/Sub delivery (push/pull), also enable the Pub/Sub API and create:
|
|
- A topic (e.g. `projects/your-project/topics/gmail-push`)
|
|
- A subscription (push and/or pull)
|
|
|
|
### Configuration
|
|
|
|
```yaml
|
|
automation:
|
|
gmail:
|
|
enabled: true
|
|
credentials_file: ~/.config/flynn/gmail-credentials.json
|
|
token_file: ~/.config/flynn/gmail-token.json # Default location
|
|
|
|
# Push mode (optional)
|
|
pubsub_topic: projects/your-project/topics/gmail-push
|
|
disable_push: false
|
|
|
|
# Pull mode (optional; no inbound webhook required)
|
|
pubsub_subscription_id: projects/your-project/subscriptions/gmail-pull
|
|
pubsub_pull_interval: "60s"
|
|
pubsub_max_messages: 10
|
|
|
|
watch_labels: [INBOX] # Labels to watch
|
|
poll_interval: "60s" # Polling fallback interval
|
|
message: "New email from {{from}}: {{subject}}\n\n{{snippet}}"
|
|
output:
|
|
channel: telegram
|
|
peer: "123456789"
|
|
```
|
|
|
|
Push notifications arrive at `POST /gmail/push` on the gateway HTTP server (bypasses gateway auth).
|
|
|
|
Pull mode uses Application Default Credentials (e.g. `GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json`) to access Pub/Sub.
|
|
|
|
### Gmail Config Fields
|
|
|
|
| Field | Required | Description |
|
|
|-------|----------|-------------|
|
|
| `enabled` | no | Enable the Gmail watcher (default: `false`) |
|
|
| `credentials_file` | yes | Path to Google OAuth2 credentials JSON |
|
|
| `token_file` | no | Path to stored OAuth2 refresh token (default: `~/.config/flynn/gmail-token.json`) |
|
|
| `pubsub_topic` | no | Pub/Sub topic for Gmail watch push notifications (`projects/<project>/topics/<topic>`) |
|
|
| `disable_push` | no | Disable watch registration even if `pubsub_topic` is set (default: `false`) |
|
|
| `pubsub_subscription_id` | no | Pub/Sub pull subscription (`projects/<project>/subscriptions/<sub>`) |
|
|
| `pubsub_pull_interval` | no | How often to pull subscription messages (default: `60s`) |
|
|
| `pubsub_max_messages` | no | Max messages per pull cycle (default: `10`) |
|
|
| `watch_labels` | no | Gmail labels to watch (default: `[INBOX]`) |
|
|
| `poll_interval` | no | Polling fallback interval: `60s`, `5m` (default: `300s`) |
|
|
| `history_start` | no | ISO date string — only process emails received after this date |
|
|
| `message` | no | Template for the agent message (default: `New email from {{from}}: {{subject}}\n\n{{snippet}}`) |
|
|
| `output.channel` | yes | Channel name to route the response (e.g. `telegram`) |
|
|
| `output.peer` | yes | Peer/chat ID on the output channel |
|
|
|
|
### Gmail Tools
|
|
|
|
When `automation.gmail.enabled: true`, Flynn registers Gmail tools:
|
|
|
|
- `gmail.list` — list recent messages by label
|
|
- `gmail.search` — query messages with Gmail search syntax
|
|
- `gmail.read` — fetch full message body by ID
|
|
- `gmail.filter.create` — create Gmail filters (criteria + actions like archive/label/forward)
|
|
|
|
`gmail.filter.create` requires a token issued with current scopes. If your token predates this change, re-run `flynn gmail-auth`.
|
|
|
|
### Template Variables
|
|
|
|
| Variable | Description |
|
|
|----------|-------------|
|
|
| `{{from}}` | Sender address |
|
|
| `{{to}}` | Recipient address |
|
|
| `{{subject}}` | Email subject line |
|
|
| `{{snippet}}` | Gmail-provided message snippet |
|
|
| `{{date}}` | Email date |
|
|
| `{{id}}` | Gmail message ID |
|
|
| `{{labels}}` | Comma-separated label names |
|
|
|
|
## Google Calendar Tools
|
|
|
|
Query Google Calendar events from within conversations. Provides three tools: `calendar.today` (today's agenda), `calendar.list` (date range), and `calendar.search` (full-text search).
|
|
|
|
### Prerequisites
|
|
|
|
1. A Google Cloud project with the **Calendar API** enabled
|
|
2. OAuth2 credentials (Desktop application type) — the same credentials file used for Gmail works
|
|
3. Run `flynn gcal-auth` to complete the OAuth2 flow and store the refresh token
|
|
- Also persisted in `~/.config/flynn/auth.json` for shared runtime refresh handling
|
|
|
|
### Configuration
|
|
|
|
```yaml
|
|
automation:
|
|
gcal:
|
|
enabled: true
|
|
credentials_file: ~/.config/flynn/gmail-credentials.json
|
|
token_file: ~/.config/flynn/gcal-token.json # Default location
|
|
calendar_ids: [primary] # Calendar IDs to query
|
|
```
|
|
|
|
### Google Calendar Config Fields
|
|
|
|
| Field | Required | Description |
|
|
|-------|----------|-------------|
|
|
| `enabled` | no | Enable the calendar tools (default: `false`) |
|
|
| `credentials_file` | yes | Path to Google OAuth2 credentials JSON |
|
|
| `token_file` | no | Path to stored OAuth2 refresh token (default: `~/.config/flynn/gcal-token.json`) |
|
|
| `calendar_ids` | no | Calendar IDs available for queries (default: `[primary]`) |
|
|
|
|
For full local operation guidance (token acquisition, storage, migration, refresh/renewal, and service scopes), see [Google OAuth Runbook](docs/operations/GOOGLE_AUTH.md).
|
|
|
|
## Google Docs, Drive, and Tasks Tools
|
|
|
|
Flynn also supports Google Docs, Drive, and Tasks tools:
|
|
|
|
- Docs: `docs.list`, `docs.search`, `docs.read`
|
|
- Drive: `drive.list`, `drive.search`, `drive.read`
|
|
- Tasks: `tasks.lists`, `tasks.list`
|
|
|
|
Enable in config:
|
|
|
|
```yaml
|
|
automation:
|
|
gdocs:
|
|
enabled: true
|
|
credentials_file: ~/.config/flynn/gmail-credentials.json
|
|
token_file: ~/.config/flynn/gdocs-token.json
|
|
gdrive:
|
|
enabled: true
|
|
credentials_file: ~/.config/flynn/gmail-credentials.json
|
|
token_file: ~/.config/flynn/gdrive-token.json
|
|
gtasks:
|
|
enabled: true
|
|
credentials_file: ~/.config/flynn/gmail-credentials.json
|
|
token_file: ~/.config/flynn/gtasks-token.json
|
|
```
|
|
|
|
Authenticate with:
|
|
|
|
```bash
|
|
flynn gdocs-auth
|
|
flynn gdrive-auth
|
|
flynn gtasks-auth
|
|
# or: flynn google-auth --service gdocs|gdrive|gtasks
|
|
```
|
|
|
|
## Vector Memory Search
|
|
|
|
The memory system supports hybrid search combining keyword matching with semantic vector similarity. When embeddings are enabled, `memory.search` uses both approaches and merges results with configurable weighting.
|
|
|
|
Memory persistence is hybrid:
|
|
- Manual writes via `memory.write`
|
|
- Automatic fact extraction during context compaction when `memory.auto_extract: true`
|
|
|
|
```yaml
|
|
memory:
|
|
enabled: true
|
|
auto_extract: true
|
|
proactive_extract:
|
|
enabled: false # Per-turn extraction beyond compaction-only writes
|
|
min_tool_calls: 1 # Trigger only when this many tool calls occur in a turn
|
|
namespace: global # Target namespace for extracted durable facts
|
|
daily_log:
|
|
enabled: false # Append each turn to dated namespaces (daily/YYYY-MM-DD)
|
|
namespace_prefix: daily
|
|
include_session_metadata: true # Include session/channel/sender headers in each log block
|
|
max_user_chars: 2000 # Per-turn cap before truncating user text
|
|
max_assistant_chars: 4000 # Per-turn cap before truncating assistant text
|
|
max_context_tokens: 2000
|
|
embedding:
|
|
enabled: true
|
|
provider: openai # openai, gemini, ollama, llamacpp
|
|
model: text-embedding-3-small
|
|
api_key: "${OPENAI_API_KEY}"
|
|
chunk_size: 512 # Tokens per chunk
|
|
chunk_overlap: 50 # Overlap between chunks
|
|
top_k: 5 # Top results from vector search
|
|
hybrid_weight: 0.7 # 0.0 = keyword only, 1.0 = vector only
|
|
qmd:
|
|
enabled: false # Experimental markdown-native search backend
|
|
top_k: 8 # Max QMD results
|
|
min_score: 0.15 # Minimum match score (0-1)
|
|
```
|
|
|
|
### Embedding Providers
|
|
|
|
| Provider | Config |
|
|
|----------|--------|
|
|
| OpenAI | `provider: openai`, `api_key`, `model` (default: `text-embedding-3-small`) |
|
|
| Gemini | `provider: gemini`, `api_key`, `model` |
|
|
| Ollama | `provider: ollama`, `endpoint` (default: localhost:11434), `model` |
|
|
| llama.cpp | `provider: llamacpp`, `endpoint`, optional `model` |
|
|
| Voyage AI | `provider: voyageai`, `api_key` or `VOYAGE_API_KEY`, `model` (default: `voyage-3-large`) |
|
|
|
|
Embeddings are indexed in the background — when memory is written, the namespace is marked dirty and re-indexed within 30 seconds. The vector index is stored in `vectors.db` alongside the session database.
|
|
|
|
Search backend selection:
|
|
|
|
- `memory.embedding.enabled: true` -> hybrid keyword+vector backend
|
|
- `memory.embedding.enabled: false` and `memory.qmd.enabled: true` -> QMD markdown backend
|
|
- otherwise -> keyword-only fallback
|
|
|
|
When the selected backend is unavailable (for example embedding provider errors), search falls back gracefully to keyword matching.
|
|
|
|
`memory.auto_extract` controls whether compaction appends extracted durable facts to `global` memory.
|
|
`memory.proactive_extract` controls optional per-turn extraction after responses (useful for tool-heavy workflows).
|
|
`memory.daily_log` controls optional append-only daily turn logs in dated namespaces. Use `include_session_metadata` when you want better cross-session traceability, and lower the per-turn char caps to reduce noise/volume in long-running chat logs.
|
|
|
|
### Proactive Context Management
|
|
|
|
Flynn can proactively signal context pressure, checkpoint conversation state to memory, and auto-compact before hard context overflows.
|
|
|
|
```yaml
|
|
compaction:
|
|
enabled: true
|
|
threshold_pct: 80
|
|
keep_turns: 4
|
|
summary_max_tokens: 1024
|
|
importance_threshold: 1
|
|
proactive:
|
|
enabled: true
|
|
warn_pct: 75
|
|
checkpoint_pct: 85
|
|
auto_compact_pct: 95
|
|
checkpoint_cooldown_ms: 300000
|
|
memory_namespace: session/checkpoints
|
|
```
|
|
|
|
Gateway surfaces this via:
|
|
- `context_warning` streamed event during `agent.send` (before `done`)
|
|
- `system.contextUsage` RPC for per-session context budget snapshots
|
|
|
|
### Embedding Config Fields
|
|
|
|
| Field | Required | Description |
|
|
|-------|----------|-------------|
|
|
| `enabled` | no | Enable vector search (default: `false`) |
|
|
| `provider` | no | Embedding provider (default: `openai`) |
|
|
| `model` | no | Embedding model name (default: `text-embedding-3-small`) |
|
|
| `endpoint` | no | Provider endpoint (required for `ollama`/`llamacpp`) |
|
|
| `api_key` | no | API key (required for `openai`/`gemini`/`voyageai`) |
|
|
| `dimensions` | no | Vector dimensions (auto-detected from model if not set) |
|
|
| `chunk_size` | no | Max tokens per chunk (default: `512`) |
|
|
| `chunk_overlap` | no | Token overlap between chunks (default: `50`) |
|
|
| `top_k` | no | Number of vector results to return (default: `5`) |
|
|
| `hybrid_weight` | no | Vector vs keyword weight, 0.0-1.0 (default: `0.7`) |
|
|
|
|
### QMD Config Fields
|
|
|
|
| Field | Required | Description |
|
|
|-------|----------|-------------|
|
|
| `enabled` | no | Enable experimental markdown-native QMD backend (default: `false`) |
|
|
| `top_k` | no | Max QMD results returned by `memory.search` (default: `8`) |
|
|
| `min_score` | no | Minimum relevance score (0.0-1.0) for QMD matches (default: `0.15`) |
|
|
|
|
## Session End Summary
|
|
|
|
Optionally summarize conversations when a WebSocket session ends and append the summary to memory.
|
|
|
|
```yaml
|
|
sessions:
|
|
end_summary:
|
|
enabled: true
|
|
tier: fast
|
|
memory_namespace: session/summaries
|
|
```
|
|
|
|
## Audit Trail
|
|
|
|
Flynn writes structured audit events to `audit.path`, including tool execution, session lifecycle, and user actions (`user.action`) from both channel and gateway requests.
|
|
|
|
Session lifecycle now includes proactive context maintenance events:
|
|
- `session.compact` for normal compaction passes
|
|
- `session.checkpoint` when proactive checkpoint summaries are written to memory
|
|
- `session.auto_compact` when proactive critical-threshold auto-compaction runs
|
|
|
|
## Gateway Lock
|
|
|
|
Single-client mode for the WebSocket gateway. When enabled, only one WebSocket connection is allowed at a time. Additional connections are rejected with close code `4003`.
|
|
|
|
```yaml
|
|
server:
|
|
lock: true
|
|
```
|
|
|
|
The web UI detects the locked state and disables auto-reconnect when rejected.
|
|
|
|
## Gateway WebSocket Rate Limit
|
|
|
|
Per-connection ingress throttling for WebSocket requests. Excess bursts are rejected; repeated violations close the connection (`4008`).
|
|
|
|
```yaml
|
|
server:
|
|
ws_rate_limit:
|
|
enabled: true
|
|
capacity: 30
|
|
refill_per_sec: 15
|
|
max_violations: 8
|
|
violation_window_ms: 10000
|
|
```
|
|
|
|
## Gateway Lane Queue Policy
|
|
|
|
Per-session FIFO queue policy for concurrent gateway requests (`agent.send`).
|
|
|
|
```yaml
|
|
server:
|
|
queue:
|
|
mode: collect # collect | followup | steer | steer_backlog | interrupt
|
|
cap: 50 # max pending requests per session lane
|
|
overflow: drop_old # drop_old | drop_new
|
|
debounce_ms: 0 # delay before running next queued item
|
|
summarize_overflow: true
|
|
overrides:
|
|
channels:
|
|
ws:
|
|
mode: followup
|
|
cap: 10
|
|
sessions:
|
|
ws:vip-user:
|
|
mode: interrupt
|
|
overflow: drop_new
|
|
debounce_ms: 100
|
|
```
|
|
|
|
Notes:
|
|
- `collect` keeps all queued requests (subject to `cap`).
|
|
- `followup` keeps at most one pending item while a request is active; newer followups replace older pending items.
|
|
- `steer` and `steer_backlog` replace pending backlog with the newest request while one is active.
|
|
- `interrupt` uses steer-backlog queueing behavior and now also requests active-run cancellation when a newer request arrives.
|
|
- When interrupt preemption occurs, gateway emits a transient content notice to the requester and writes a `queue.preempt` audit event.
|
|
- Active cancellation remains best-effort and stops at agent safe points; use `agent.cancel` for explicit user-triggered cancellation control.
|
|
- `debounce_ms` delays the next queued execution, helping collapse bursty same-session traffic.
|
|
- `summarize_overflow` enables richer overflow error messages and payload metadata.
|
|
- On overflow, `drop_old` evicts the oldest pending request, `drop_new` rejects the new request.
|
|
- Override precedence: exact `sessions` match first, then `channels`.
|
|
|
|
Runtime session controls from chat commands:
|
|
- `/queue` shows effective session queue policy.
|
|
- `/queue set <mode|cap|overflow|debounce_ms|summarize_overflow> <value>` sets a per-session override.
|
|
- `/queue reset` clears per-session queue overrides.
|
|
|
|
## Gateway Node Capability Negotiation
|
|
|
|
Optional gateway surface for companion clients and node-role negotiation:
|
|
|
|
```yaml
|
|
server:
|
|
nodes:
|
|
enabled: true
|
|
allowed_roles: [companion]
|
|
feature_gates:
|
|
ui.canvas: true
|
|
location:
|
|
enabled: true
|
|
push:
|
|
enabled: true
|
|
```
|
|
|
|
Methods:
|
|
- `node.register` registers role + declared capabilities for the current connection.
|
|
- `node.capabilities.get` returns negotiated protocol version and enabled capabilities.
|
|
- `node.location.set` updates the node's last-known location (when `server.nodes.location.enabled` is true).
|
|
- `node.location.get` returns the node's stored location payload.
|
|
- `node.status.set` publishes companion status/heartbeat fields (`platform`, `appVersion`, `batteryPct`, etc.).
|
|
- `node.push_token.set` registers node push tokens (APNs for iOS/macOS, FCM for Android) when `server.nodes.push.enabled` is true.
|
|
- `system.location` provides an operator view of registered node locations.
|
|
- `system.nodes` returns registered node snapshots (role, capabilities, identity, location/status).
|
|
- `system.capabilities` returns gateway protocol and node policy snapshot.
|
|
|
|
Companion runtime helper:
|
|
- `src/companion/runtimeClient.ts` provides a typed Node/WebSocket client for companion runtimes (macOS/iOS/Android workers) with wrappers for `node.register`, `node.capabilities.get`, `node.location.set/get`, `node.status.set`, `node.push_token.set`, `system.capabilities`, `system.nodes`, and canvas artifact RPCs (`canvas.put/get/list/delete/clear`), plus convenience helpers (`bootstrapNode`, optional `autoConnect`, `dispose()`, `waitForIdle()` for pending-work drain synchronization) and event helpers (`subscribeEvents()`, `subscribeEvent()`, `subscribeAgentStream()`, `subscribeAgentTyping()`, `subscribeContextWarning()`, `waitForEvent()` with timeout/predicate/abort support plus event-name/timeout validation and deterministic teardown cancellation including socket-close rejection, `waitForAnyEvent()` with event-list/timeout validation, `waitForAgentStream()`, `waitForAgentTyping()`, `waitForContextWarning()`, `clearEventSubscriptions()` returning cleared-subscription/cancelled-wait counts, `cancelPendingEventWaits()` returning cancelled waiter count, `listKnownEventNames()`, `eventSubscriptionCount`) plus in-flight observability via `pendingRequestCount`, `pendingEventWaitCount`, `hasPendingWork`, `idle`, `lastDisconnectCode`, `lastDisconnectReason`, `getPendingWorkSnapshot()`, `getEventSurfaceSnapshot()`, and `getConnectionSnapshot()` (including disconnect metadata, cleared on successful reconnect).
|
|
- `src/companion/platformClients.ts` provides platform-focused wrappers:
|
|
- `MacOSCompanionClient` (`platform: "macos"`, APNs push registration)
|
|
- `IOSCompanionClient` (`platform: "ios"`, APNs push registration)
|
|
- `AndroidCompanionClient` (`platform: "android"`, FCM push registration)
|
|
- shared `bootstrap()` helper (`register` + `getCapabilities`, optional `system.capabilities`) for startup handshakes
|
|
- shared `publishHeartbeat()` helper for periodic `node.status.set` updates with safe defaults
|
|
- `createHeartbeatLoop()` convenience helper that returns a bound `CompanionHeartbeatLoop`
|
|
- optional `defaultSessionId` for canvas helper calls so `sessionId` can be omitted per call
|
|
- lifecycle passthroughs for connection state/teardown (`connected`, `disconnect(code?, reason?)`, `dispose(code?, reason?)`)
|
|
- stream passthrough helpers (`subscribeEvents`, `subscribeEvent`, `clearEventSubscriptions`, `cancelPendingEventWaits`, `listKnownEventNames`, `eventSubscriptionCount`, `subscribeAgentStream/Typing/ContextWarning`, `waitForEvent`, `waitForAnyEvent`, `waitForAgentStream/Typing/ContextWarning`)
|
|
- runtime observability/control passthroughs (`pendingRequestCount`, `pendingEventWaitCount`, `hasPendingWork`, `idle`, `lastDisconnectCode`, `lastDisconnectReason`, `getPendingWorkSnapshot()`, `getEventSurfaceSnapshot()`, `getConnectionSnapshot()`, `connected`, `waitForIdle()`)
|
|
- `src/companion/heartbeatLoop.ts` provides `CompanionHeartbeatLoop` for periodic heartbeat scheduling (`publishHeartbeat`) with start/stop safety, optional interval jitter (`jitterRatio`) to spread load (with safe normalization for invalid random samples), `tickNow()` for manual sends, success/error hooks, loop observability (`successCount`, `lastSuccessAt`, `failureCount`, `lastFailure`, `getState()`), and optional auto-stop after repeated failures.
|
|
|
|
Minimal companion CLI:
|
|
- `flynn companion --once` connects to the gateway, registers a node, publishes one heartbeat, then exits.
|
|
- `flynn companion --platform macos --heartbeat 30` runs a long-lived node with periodic heartbeats and logs `agent.stream`/`agent.typing` events.
|
|
|
|
## WebChat PWA Push Subscriptions
|
|
|
|
Enable installable WebChat PWA metadata and browser push-subscription storage on the gateway:
|
|
|
|
```yaml
|
|
server:
|
|
webchat_push:
|
|
enabled: true
|
|
vapid_public_key: ${WEBCHAT_VAPID_PUBLIC_KEY}
|
|
max_subscriptions: 5000
|
|
```
|
|
|
|
Notes:
|
|
- WebChat now serves `manifest.webmanifest` and a service worker (`/sw.js`).
|
|
- Settings page includes Push controls (Enable/Disable) that subscribe the current browser and register/unregister its endpoint with the gateway.
|
|
- Gateway endpoints:
|
|
- `GET /webchat/push/public-key`
|
|
- `GET /webchat/push/subscriptions`
|
|
- `POST /webchat/push/subscriptions`
|
|
- `DELETE /webchat/push/subscriptions`
|
|
- These endpoints are protected by normal gateway HTTP auth (`server.token` + `server.auth_http`) and support `?token=` query auth for browser clients.
|
|
|
|
## Canvas / A2UI Foundation
|
|
|
|
Gateway provides a session-scoped canvas artifact API for companion/UI surfaces:
|
|
- `canvas.put` upserts an artifact (`artifactId`, `type`, `content`, optional `title`/`metadata`).
|
|
- `canvas.get` retrieves a single artifact.
|
|
- `canvas.list` lists artifacts for a session (most recently updated first).
|
|
- `canvas.delete` removes one artifact.
|
|
- `canvas.clear` removes all artifacts for a session.
|
|
|
|
This foundation is currently in-memory (runtime ephemeral) and intended as the first step for richer visual workspace flows.
|
|
|
|
## Gateway Request Body Limit
|
|
|
|
Cap inbound HTTP POST body size (webhooks and Gmail push) to reduce memory-DoS risk.
|
|
|
|
```yaml
|
|
server:
|
|
max_request_body_bytes: 1048576 # 1 MiB
|
|
```
|
|
|
|
## Tailscale Serve
|
|
|
|
Automatically expose the gateway via Tailscale Serve when the daemon starts. Requires Tailscale to be installed and authenticated on the host.
|
|
|
|
```yaml
|
|
server:
|
|
tailscale:
|
|
serve: true
|
|
```
|
|
|
|
When enabled, Flynn runs `tailscale serve` on startup to expose the gateway port over your tailnet, and cleans up on shutdown. The `flynn doctor` command includes a Tailscale availability check.
|
|
|
|
## Bonjour / mDNS Discovery
|
|
|
|
Optionally advertise the gateway on your local network so LAN clients can discover Flynn without manual host entry.
|
|
|
|
```yaml
|
|
server:
|
|
localhost: false
|
|
discovery:
|
|
enabled: true
|
|
service_name: flynn-gateway
|
|
service_type: _flynn._tcp
|
|
txt:
|
|
env: home-lab
|
|
```
|
|
|
|
Notes:
|
|
- Discovery is disabled by default.
|
|
- `server.localhost` must be `false` for LAN clients to connect.
|
|
- Flynn advertises non-secret metadata only (instance/version + optional `txt` keys you provide).
|
|
- Runtime uses host tools (`avahi-publish-service` on Linux, `dns-sd` on macOS) when available.
|
|
|
|
## DM Pairing Codes
|
|
|
|
Allow unknown senders to authenticate with the bot via time-limited pairing codes. Works across all channel adapters (Telegram, Discord, Slack, WhatsApp).
|
|
|
|
```yaml
|
|
pairing:
|
|
enabled: true
|
|
code_ttl: "10m" # Code expiry time (default: 10 minutes)
|
|
code_length: 6 # Code length (default: 6 digits)
|
|
```
|
|
|
|
### How it works
|
|
|
|
1. Generate a code via the TUI (`/pair generate`), gateway API (`pairing.generate`), or web dashboard
|
|
2. Share the code with the user
|
|
3. The user sends the code as their first DM to the bot
|
|
4. If valid, the user's sender ID is permanently approved for that channel (persisted in SQLite, survives daemon restarts)
|
|
5. Approved users can be listed (`/pair list`) and revoked (`/pair revoke <channel> <id>`)
|
|
|
|
### TUI Commands
|
|
|
|
| Command | Description |
|
|
|---------|-------------|
|
|
| `/pair` | Generate a new pairing code |
|
|
| `/pair generate [label]` | Generate a code with optional label |
|
|
| `/pair list` | List pending codes and approved senders |
|
|
| `/pair revoke <channel> <id>` | Revoke an approved sender |
|
|
|
|
## WhatsApp Chromium Sandbox
|
|
|
|
WhatsApp adapter now launches Chromium in sandboxed mode by default.
|
|
|
|
If you must disable Chromium sandboxing in a high-trust/containerized environment:
|
|
|
|
```yaml
|
|
whatsapp:
|
|
no_sandbox: true
|
|
```
|
|
|
|
### Gateway API
|
|
|
|
| Method | Description |
|
|
|--------|-------------|
|
|
| `pairing.generate` | Generate a new pairing code (optional `label` param) |
|
|
| `pairing.list` | List pending codes and approved senders |
|
|
| `pairing.revoke` | Revoke an approved sender (`channel` + `senderId` params) |
|
|
| `system.presence` | List observed sender presence with online/offline status inferred from recent inbound activity |
|
|
|
|
## Shell Completion
|
|
|
|
Generate shell completions for bash, zsh, or fish:
|
|
|
|
```bash
|
|
# Print completions to stdout
|
|
flynn completion bash
|
|
flynn completion zsh
|
|
flynn completion fish
|
|
|
|
# Install directly to shell config
|
|
flynn completion bash --install # Appends to ~/.bashrc
|
|
flynn completion zsh --install # Appends to ~/.zshrc
|
|
flynn completion fish --install # Writes to ~/.config/fish/completions/flynn.fish
|
|
```
|
|
|
|
## Docker Deployment
|
|
|
|
Flynn includes a production-ready Dockerfile with multi-stage build.
|
|
|
|
```bash
|
|
# Build the image
|
|
docker build -t flynn .
|
|
|
|
# Run with config and data volumes
|
|
docker run -d \
|
|
--name flynn \
|
|
-p 18800:18800 \
|
|
-v ./config.yaml:/config/config.yaml:ro \
|
|
-v flynn-data:/data \
|
|
-e ANTHROPIC_API_KEY=sk-ant-... \
|
|
flynn
|
|
```
|
|
|
|
Or use the included `docker-compose.yml`:
|
|
|
|
```bash
|
|
# Copy your config
|
|
cp ~/.config/flynn/config.yaml ./config.yaml
|
|
|
|
# Start with compose
|
|
docker compose up -d
|
|
|
|
# View logs
|
|
docker compose logs -f
|
|
```
|
|
|
|
### Docker Environment Variables
|
|
|
|
| Variable | Description |
|
|
|----------|-------------|
|
|
| `FLYNN_CONFIG` | Config file path (default: `/config/config.yaml`) |
|
|
| `FLYNN_DATA_DIR` | Data directory path (default: `/data`) |
|
|
| `ANTHROPIC_API_KEY` | Anthropic API key |
|
|
| `OPENAI_API_KEY` | OpenAI API key |
|
|
| `TELEGRAM_BOT_TOKEN` | Telegram bot token |
|
|
|
|
### Volumes
|
|
|
|
| Mount Point | Purpose |
|
|
|-------------|---------|
|
|
| `/config/config.yaml` | Configuration file (read-only) |
|
|
| `/data` | Persistent data (sessions DB, memory files, vector index) |
|
|
|
|
## PaaS Deployment (Fly.io / Railway / Render)
|
|
|
|
See `docs/deployment/PAAS.md` and the templates under `deploy/`.
|
|
|
|
## Nix Deployment
|
|
|
|
See `docs/deployment/NIX.md` for the flake (package + dev shell + optional NixOS module).
|
|
|
|
## Doctor Diagnostics
|
|
|
|
`flynn doctor` runs health checks to validate your setup:
|
|
|
|
```
|
|
$ flynn doctor
|
|
|
|
Flynn Doctor
|
|
============
|
|
|
|
[PASS] Config file exists (/home/user/.config/flynn/config.yaml)
|
|
[PASS] Config parses (valid YAML)
|
|
[PASS] Config validates (schema valid)
|
|
[PASS] Env vars resolved
|
|
[PASS] Data directory writable (/home/user/.local/share/flynn)
|
|
[PASS] Session DB accessible (sessions.db)
|
|
[PASS] Model connectivity (anthropic: claude-sonnet)
|
|
[PASS] Telegram bot configured (1 allowed chat(s))
|
|
[SKIP] MCP servers configured (none configured)
|
|
[PASS] Skills loaded (3 skill(s))
|
|
|
|
Results: 8 passed, 0 failed, 0 warnings, 1 skipped
|
|
```
|
|
|
|
### Check Details
|
|
|
|
| Check | What it validates |
|
|
|-------|-------------------|
|
|
| Config file exists | Config YAML file is present at the expected path |
|
|
| Config parses | File is valid YAML syntax |
|
|
| Config validates | YAML content passes Zod schema validation |
|
|
| Env vars resolved | Any `${VAR}` references in config have values set |
|
|
| Data directory writable | Can write to `~/.local/share/flynn/` |
|
|
| Session DB accessible | SQLite database opens and queries succeed |
|
|
| Model connectivity | Default model provider and model name are configured |
|
|
| Telegram bot configured | Bot token is present and reasonable length |
|
|
| MCP servers configured | Lists configured MCP tool servers |
|
|
| Skills loaded | Discovers and loads skill packages |
|
|
| Skills registry | Verifies registry source is configured and catalog is reachable/parsible |
|
|
|
|
Exit code is `1` if any check fails, `0` otherwise. Checks that depend on a valid config are skipped when config is invalid.
|
|
|
|
## Session Management
|
|
|
|
- Sessions persist in `~/.local/share/flynn/sessions.db`
|
|
- Session ID format: `{frontend}:{userId}` (e.g., `telegram:123456789`)
|
|
- History survives restarts
|
|
- Transfer sessions between frontends with `/transfer <telegram|tui>`
|
|
|
|
## Architecture
|
|
|
|
```
|
|
src/
|
|
├── agents/ # Multi-agent routing
|
|
├── auth/ # OAuth flows (GitHub Copilot)
|
|
├── backends/native/ # Agent implementation + orchestrator
|
|
├── channels/ # Channel adapters (Telegram, Discord, Slack, WhatsApp, Matrix, WebChat)
|
|
│ └── pairing.ts # DM pairing code manager
|
|
├── cli/ # CLI commands (commander)
|
|
│ └── completion.ts # Shell completion generator (bash/zsh/fish)
|
|
├── config/ # YAML config + Zod validation
|
|
├── context/ # Token estimation + compaction
|
|
├── daemon/ # Lifecycle management + routing
|
|
├── frontends/
|
|
│ ├── telegram/ # Telegram bot
|
|
│ └── tui/ # Terminal UI (minimal + fullscreen)
|
|
├── gateway/ # WebSocket gateway + web UI dashboard
|
|
│ ├── handlers/ # JSON-RPC method handlers (agent, sessions, system, pairing, tools, config)
|
|
│ ├── tailscale.ts # Tailscale Serve lifecycle management
|
|
│ ├── lane-queue.ts # Per-session FIFO request queue
|
|
│ └── ui/ # SPA dashboard (vanilla JS)
|
|
│ ├── pages/ # Dashboard, Chat, Sessions, Usage, Settings
|
|
│ └── lib/ # WebSocket RPC client
|
|
├── hooks/ # Confirmation engine
|
|
├── mcp/ # MCP tool server integration
|
|
├── memory/ # Persistent memory store + vector search
|
|
├── models/ # Model providers + router + media pipeline + audio capabilities
|
|
├── prompt/ # System prompt templating (auto-injects current date/time)
|
|
├── sandbox/ # Docker sandboxing
|
|
├── session/ # SQLite persistence
|
|
├── skills/ # Skill packages
|
|
├── tools/ # Builtin tools (shell, file, web, browser, process, media, audio, system.info)
|
|
└── automation/ # Cron scheduler, webhooks, heartbeat monitor, Gmail watcher
|
|
```
|
|
|
|
## System Prompt
|
|
|
|
Flynn assembles its system prompt from layered template files (`SOUL.md`, `AGENTS.md`, `IDENTITY.md`, `USER.md`, `TOOLS.md`, `BOOTSTRAP.md`, `HEARTBEAT.md`) searched in configurable directories. The first match per file wins.
|
|
|
|
A **Runtime Context** section is automatically appended to every system prompt with the current date and time, so the model always knows when "now" is without needing a tool call.
|
|
|
|
The `system.info` tool provides on-demand access to more detailed runtime information: current date/time, hostname, platform, architecture, OS release, uptime, Node.js version, memory usage, and working directory.
|
|
|
|
## Development
|
|
|
|
```bash
|
|
# Dev mode with watch
|
|
pnpm dev # Daemon
|
|
pnpm tui:dev # TUI
|
|
|
|
# Type check
|
|
pnpm typecheck
|
|
|
|
# Lint
|
|
pnpm lint
|
|
|
|
# Test
|
|
pnpm test
|
|
```
|
|
|
|
## Environment Variables
|
|
|
|
| Variable | Description |
|
|
|----------|-------------|
|
|
| `FLYNN_CONFIG` | Override config file path |
|
|
| `FLYNN_DATA_DIR` | Override data directory (default: `~/.local/share/flynn`) |
|
|
| `ANTHROPIC_API_KEY` | Anthropic API key (fallback) |
|
|
| `OPENAI_API_KEY` | OpenAI API key (fallback) |
|
|
| `GEMINI_API_KEY` | Gemini API key (fallback; `GOOGLE_API_KEY` also supported) |
|
|
|
|
## License
|
|
|
|
MIT
|