docs: map existing codebase
This commit is contained in:
William Valentin
@@ -0,0 +1,323 @@
|
||||
# External Integrations
|
||||
|
||||
**Analysis Date:** 2026-02-09
|
||||
|
||||
## AI Model Providers
|
||||
|
||||
Flynn supports 10 model providers via a unified `ModelClient` interface (`src/models/types.ts`). Each provider implements `chat()` and optionally `chatStream()`. The `ModelRouter` (`src/models/router.ts`) manages tier-based routing (fast/default/complex/local) with fallback chains.
|
||||
|
||||
**Anthropic:**
|
||||
- SDK: `@anthropic-ai/sdk` (`src/models/anthropic.ts`)
|
||||
- Auth: `ANTHROPIC_API_KEY` env var or `api_key` in config
|
||||
- Features: Streaming, tool use, extended thinking mode, multimodal (images)
|
||||
- Extended thinking: `{ type: 'enabled', budget_tokens: 4096 }` on request
|
||||
|
||||
**OpenAI:**
|
||||
- SDK: `openai` (`src/models/openai.ts`)
|
||||
- Auth: `OPENAI_API_KEY` env var or `api_key` in config
|
||||
- Features: Tool use, multimodal (images via data URIs or URLs)
|
||||
- Also powers: OpenRouter, ZhipuAI, xAI via `baseURL` override
|
||||
|
||||
**Google Gemini:**
|
||||
- SDK: `@google/generative-ai` (`src/models/gemini.ts`)
|
||||
- Auth: `GOOGLE_API_KEY` env var or `api_key` in config
|
||||
- Features: Streaming, tool use, extended thinking, multimodal
|
||||
|
||||
**AWS Bedrock:**
|
||||
- SDK: `@aws-sdk/client-bedrock-runtime` (`src/models/bedrock.ts`)
|
||||
- Auth: `AWS_REGION` env var + IAM credentials or explicit `accessKeyId`/`secretAccessKey` in config
|
||||
- Features: Streaming (ConverseStream), tool use, multimodal
|
||||
- Models: Meta Llama, Amazon Titan (cost-tracked in `src/models/costs.ts`)
|
||||
|
||||
**GitHub Models (Copilot):**
|
||||
- SDK: `openai` (OpenAI-compatible API) (`src/models/github.ts`)
|
||||
- Auth: `GITHUB_TOKEN` env var or OAuth device flow (`src/auth/github.ts`)
|
||||
- Endpoint: `https://api.githubcopilot.com`
|
||||
- Auto-fallback: When an Anthropic tier fails, Flynn automatically tries the same model via GitHub Models before the global fallback chain (`src/daemon/index.ts` `createAutoFallbackClient()`)
|
||||
- OAuth device flow: Uses client ID `Ov23li8tweQw6odWQebz`, stores token at `~/.config/flynn/auth.json`
|
||||
|
||||
**OpenRouter:**
|
||||
- SDK: `openai` with `baseURL: https://openrouter.ai/api/v1` (`src/daemon/index.ts`)
|
||||
- Auth: `OPENROUTER_API_KEY` env var or `api_key` in config
|
||||
|
||||
**ZhipuAI:**
|
||||
- SDK: `openai` with `baseURL: https://api.z.ai/api/paas/v4` (`src/daemon/index.ts`)
|
||||
- Auth: `ZHIPUAI_API_KEY` env var or `api_key` in config
|
||||
|
||||
**xAI (Grok):**
|
||||
- SDK: `openai` with `baseURL: https://api.x.ai/v1` (`src/daemon/index.ts`)
|
||||
- Auth: `XAI_API_KEY` env var or `api_key` in config
|
||||
|
||||
**Ollama (Local):**
|
||||
- SDK: `ollama` (`src/models/local/ollama.ts`)
|
||||
- Auth: None (local server)
|
||||
- Endpoint: Configurable `host` (default: `http://localhost:11434`)
|
||||
- Config: `num_gpu` option for GPU layer control
|
||||
|
||||
**llama.cpp (Local):**
|
||||
- SDK: Raw `fetch` HTTP calls (`src/models/local/llamacpp.ts`)
|
||||
- Auth: Optional `auth_token` header
|
||||
- Endpoint: Configurable (default: `http://localhost:8080`)
|
||||
|
||||
## Embedding Providers
|
||||
|
||||
Embedding providers (`src/memory/embeddings.ts`) power the hybrid vector + keyword search system. Factory function: `createEmbeddingProvider()`.
|
||||
|
||||
**OpenAI Embeddings:**
|
||||
- SDK: `openai` (lazy import)
|
||||
- Auth: `OPENAI_API_KEY` or config `api_key`
|
||||
- Default model: `text-embedding-3-small`, default dims: 1536
|
||||
|
||||
**Gemini Embeddings:**
|
||||
- SDK: `@google/generative-ai` (lazy import)
|
||||
- Auth: `GOOGLE_API_KEY` or config `api_key`
|
||||
- Uses `batchEmbedContents` for efficiency, default dims: 768
|
||||
|
||||
**Ollama Embeddings:**
|
||||
- SDK: `ollama` (lazy import)
|
||||
- Auth: None (local)
|
||||
- Configurable host endpoint, default dims: 768
|
||||
|
||||
**LlamaCpp Embeddings:**
|
||||
- SDK: Raw `fetch` to `/embedding` endpoint
|
||||
- Auth: None
|
||||
- Default endpoint: `http://localhost:8080`, default dims: 768
|
||||
|
||||
**Voyage AI Embeddings:**
|
||||
- SDK: `openai` (OpenAI-compatible API, lazy import)
|
||||
- Auth: `VOYAGE_API_KEY` env var or config `api_key`
|
||||
- Endpoint: `https://api.voyageai.com/v1`, default dims: 1024
|
||||
|
||||
## Data Storage
|
||||
|
||||
**Session Database (SQLite):**
|
||||
- Library: `better-sqlite3` (`src/session/store.ts`)
|
||||
- Location: `{dataDir}/sessions.db`
|
||||
- Schema: `messages` table with `id`, `session_id`, `role`, `content`, `created_at`
|
||||
- TTL-based pruning: Configurable via `sessions.ttl` (default: 30 days), hourly cleanup
|
||||
|
||||
**Vector Database (SQLite):**
|
||||
- Library: `better-sqlite3` (`src/memory/vector-store.ts`)
|
||||
- Location: `{dataDir}/vectors.db`
|
||||
- Stores embedding chunks as `Float32Array` BLOBs
|
||||
- Content hashing for deduplication
|
||||
- Background indexer runs every 30 seconds
|
||||
|
||||
**Memory Store (Filesystem):**
|
||||
- Location: `{dataDir}/memory/` (`src/memory/store.ts`)
|
||||
- Format: Markdown files organized by namespace
|
||||
- Layout: `global.md`, `user.md`, `sessions/{id}.md`
|
||||
- Hybrid search: Keyword + vector (configurable weight via `hybrid_weight`, default 0.7)
|
||||
|
||||
**File Storage:**
|
||||
- Local filesystem only — no cloud object storage
|
||||
|
||||
**Caching:**
|
||||
- In-memory response cache for web fetch tool (5-minute TTL) (`src/tools/builtin/web-fetch.ts`)
|
||||
- No external cache service (Redis, etc.)
|
||||
|
||||
## Channel Adapters (Messaging Platforms)
|
||||
|
||||
All adapters implement `ChannelAdapter` interface (`src/channels/types.ts`): `connect()`, `disconnect()`, `send()`, `onMessage()`.
|
||||
|
||||
**Telegram:**
|
||||
- SDK: `grammy` (`src/channels/telegram/`)
|
||||
- Auth: Bot token via `telegram.bot_token` config
|
||||
- Features: Long polling, chat ID allowlist, mention requirement, pairing codes, image/audio attachments
|
||||
|
||||
**Discord:**
|
||||
- SDK: `discord.js` (`src/channels/discord/`)
|
||||
- Auth: Bot token via `discord.bot_token` config
|
||||
- Features: Guild/channel allowlists, mention requirement, pairing codes
|
||||
|
||||
**Slack:**
|
||||
- SDK: `@slack/bolt` (`src/channels/slack/`)
|
||||
- Auth: `bot_token`, `app_token`, `signing_secret` in config
|
||||
- Features: Socket mode, channel allowlists, mention requirement, pairing codes
|
||||
|
||||
**WhatsApp:**
|
||||
- SDK: `whatsapp-web.js` (`src/channels/whatsapp/`)
|
||||
- Auth: QR code scanning (web client emulation)
|
||||
- Features: Number/group allowlists, mention requirement, custom data directory, pairing codes
|
||||
|
||||
**WebChat:**
|
||||
- Implementation: Gateway WebSocket bridge (`src/channels/webchat/`)
|
||||
- Auth: Gateway token or Tailscale identity
|
||||
- UI: Vanilla JS dashboard at `src/gateway/ui/` (HTML + CSS + JS, no framework)
|
||||
|
||||
## Authentication & Identity
|
||||
|
||||
**GitHub OAuth (Device Flow):**
|
||||
- Implementation: `src/auth/github.ts`
|
||||
- Client ID: `Ov23li8tweQw6odWQebz` (GitHub Copilot)
|
||||
- Flow: Device code → User authorization → Token polling
|
||||
- Storage: `~/.config/flynn/auth.json` (600 permissions)
|
||||
- Priority: `GITHUB_TOKEN` env → stored OAuth token → `null`
|
||||
|
||||
**Gateway Auth:**
|
||||
- Static bearer token (`server.token` in config)
|
||||
- Tailscale identity header trust (`server.tailscale_identity`)
|
||||
- HTTP auth optional (`server.auth_http`)
|
||||
- Gateway lock: Single-client WebSocket mode (`server.lock`)
|
||||
|
||||
**DM Pairing Codes:**
|
||||
- Implementation: `src/channels/pairing.ts`
|
||||
- Purpose: Authenticate unknown senders via one-time codes
|
||||
- Config: `pairing.enabled`, `pairing.code_ttl` (default 5m), `pairing.code_length` (default 6)
|
||||
- Gateway handlers for code generation/verification
|
||||
|
||||
**Gmail OAuth2:**
|
||||
- SDK: `googleapis` (`src/automation/gmail.ts`)
|
||||
- Credentials: `~/.config/flynn/gmail-credentials.json`
|
||||
- Token: `~/.config/flynn/gmail-token.json`
|
||||
- Setup: `flynn gmail-auth` CLI command
|
||||
|
||||
## Automation
|
||||
|
||||
**Cron Scheduler:**
|
||||
- Library: `croner` (`src/automation/cron.ts`)
|
||||
- Config: `automation.cron[]` — each job has `name`, `schedule`, `message`, `output.channel`, `output.peer`
|
||||
- Implements `ChannelAdapter` to inject cron-triggered messages into the channel registry
|
||||
- Features: Enable/disable per job, timezone support, runtime management tools
|
||||
|
||||
**Webhooks:**
|
||||
- Implementation: `src/automation/webhooks.ts`
|
||||
- Auth: HMAC-SHA256 signature verification (`X-Webhook-Signature` header)
|
||||
- Templates: `{{body}}` and `{{json.field}}` placeholders
|
||||
- Route: `POST /webhooks/{name}` on the gateway HTTP server
|
||||
- Config: `automation.webhooks[]` with `name`, `secret`, `message`, `output`
|
||||
|
||||
**Gmail Watcher:**
|
||||
- SDK: `googleapis` (`src/automation/gmail.ts`)
|
||||
- Modes: Pub/Sub push notifications or polling fallback
|
||||
- Pub/Sub topic: `projects/flynn-agent/topics/gmail-push`
|
||||
- Watch renewal: Every 6 days (Google watch expires at ~7 days)
|
||||
- Config: `automation.gmail` with `watch_labels`, `poll_interval`, `history_start`
|
||||
- Route: `POST /gmail/push` on gateway for Pub/Sub push
|
||||
|
||||
**Heartbeat Monitor:**
|
||||
- Implementation: `src/automation/heartbeat.ts`
|
||||
- Checks: gateway, model, channels, memory, disk
|
||||
- Config: `automation.heartbeat` with `interval`, `checks`, `failure_threshold`, `disk_threshold_mb`
|
||||
- Notification: Sends to configured channel/peer on failures
|
||||
|
||||
## Web & Content Tools
|
||||
|
||||
**Web Search (Brave / SearXNG):**
|
||||
- Implementation: `src/tools/builtin/web-search.ts`
|
||||
- Brave Search API: `https://api.search.brave.com/res/v1/web/search`
|
||||
- Auth: `X-Subscription-Token` header via `web_search.api_key`
|
||||
- SearXNG: Self-hosted instance via `web_search.endpoint`
|
||||
- Auth: None (private instance)
|
||||
- Config: `web_search.provider` (`brave` or `searxng`), `web_search.max_results`
|
||||
|
||||
**Web Fetch (Readability):**
|
||||
- Libraries: `linkedom`, `@mozilla/readability`, `turndown` (`src/tools/builtin/web-fetch.ts`)
|
||||
- Features: HTML → Markdown conversion, article extraction, response caching (5min TTL)
|
||||
- Truncation: 50,000 character max
|
||||
|
||||
**Browser Automation:**
|
||||
- Library: `puppeteer-core` (`src/tools/builtin/browser/`)
|
||||
- Config: `browser.executable_path` or `browser.ws_endpoint`
|
||||
- Features: Headless browsing, page management, screenshots
|
||||
- Limits: `browser.max_pages` (default 5), `browser.default_timeout` (default 30s)
|
||||
|
||||
## Audio Transcription
|
||||
|
||||
**Whisper-Compatible API:**
|
||||
- Implementation: `src/models/media.ts`
|
||||
- Endpoint: Configurable via `audio.transcription_endpoint`
|
||||
- Auth: `audio.transcription_api_key` (Bearer token)
|
||||
- Model: `audio.transcription_model` (default: `whisper-1`)
|
||||
- Supported formats: OGG, MP3, WAV, WebM, MP4, M4A
|
||||
- Integration: Auto-transcribes audio attachments from channels before model processing
|
||||
|
||||
## MCP (Model Context Protocol)
|
||||
|
||||
**MCP Client:**
|
||||
- SDK: `@modelcontextprotocol/sdk` (`src/mcp/client.ts`)
|
||||
- Transport: stdio (spawns external processes)
|
||||
- Config: `mcp.servers[]` with `name`, `command`, `args`, `env`, `cwd`
|
||||
- Bridge: MCP tools auto-registered in Flynn's tool registry (`src/mcp/bridge.ts`)
|
||||
- Management: `McpManager` starts/stops all configured servers (`src/mcp/manager.ts`)
|
||||
|
||||
## Docker Sandbox
|
||||
|
||||
**Per-Session Containers:**
|
||||
- Implementation: `src/sandbox/manager.ts`, `src/sandbox/docker.ts`
|
||||
- Config: `sandbox.image` (default: `node:22-slim`), `sandbox.network` (default: `none`), `sandbox.memory_limit`, `sandbox.cpu_limit`
|
||||
- Features: Lazily created per session, replaces `shell.exec` and `process.start` tools with sandboxed versions
|
||||
- Prerequisite: Docker daemon available
|
||||
|
||||
## Networking & Exposure
|
||||
|
||||
**Gateway Server:**
|
||||
- Protocol: WebSocket (JSON-RPC) + HTTP (`src/gateway/server.ts`)
|
||||
- Default port: 18800
|
||||
- Binding: `127.0.0.1` (localhost only) or `0.0.0.0`
|
||||
- Features: LaneQueue for request ordering, session bridge, static file serving for dashboard
|
||||
|
||||
**Tailscale Serve:**
|
||||
- Implementation: `src/gateway/tailscale.ts`
|
||||
- Purpose: Expose gateway HTTPS endpoint on tailnet
|
||||
- Config: `server.tailscale.serve`, `server.tailscale.hostname`, `server.tailscale.port`
|
||||
- Prerequisite: Tailscale CLI installed and daemon running
|
||||
|
||||
## Monitoring & Observability
|
||||
|
||||
**Error Tracking:**
|
||||
- None (console.error only)
|
||||
|
||||
**Logging:**
|
||||
- `console.log` / `console.error` / `console.debug` throughout
|
||||
- No structured logging framework
|
||||
|
||||
**Cost Tracking:**
|
||||
- Built-in: `src/models/costs.ts` with per-million-token pricing for known models
|
||||
- Tracks: Anthropic, OpenAI, Gemini, xAI, Bedrock models
|
||||
- GitHub Copilot models tracked at $0 (subscription-included)
|
||||
- Usage exposed via `/usage` command and gateway `system.usage` RPC
|
||||
|
||||
## CI/CD & Deployment
|
||||
|
||||
**Hosting:**
|
||||
- Self-hosted (designed for personal deployment)
|
||||
- Process supervisor expected for restarts (exit code 75 = restart signal)
|
||||
|
||||
**CI Pipeline:**
|
||||
- Not detected in repository
|
||||
|
||||
## Environment Configuration
|
||||
|
||||
**Required env vars (minimum viable):**
|
||||
- `ANTHROPIC_API_KEY` (or other model provider key)
|
||||
- `FLYNN_TELEGRAM_TOKEN` (if using default Telegram channel)
|
||||
|
||||
**Optional env vars (by feature):**
|
||||
- `OPENAI_API_KEY` - OpenAI models and embeddings
|
||||
- `GOOGLE_API_KEY` - Gemini models and embeddings
|
||||
- `GITHUB_TOKEN` - GitHub Models / Copilot access
|
||||
- `AWS_REGION` - Bedrock region
|
||||
- `OPENROUTER_API_KEY` - OpenRouter access
|
||||
- `ZHIPUAI_API_KEY` - ZhipuAI access
|
||||
- `XAI_API_KEY` - xAI (Grok) access
|
||||
- `VOYAGE_API_KEY` - Voyage AI embeddings
|
||||
- `FLYNN_DATA_DIR` - Custom data directory
|
||||
|
||||
**Secrets location:**
|
||||
- API keys: YAML config (with `${ENV_VAR}` expansion) or environment variables
|
||||
- OAuth tokens: `~/.config/flynn/auth.json` (GitHub), `~/.config/flynn/gmail-token.json` (Gmail)
|
||||
- `.env.example` present at project root
|
||||
|
||||
## Webhooks & Callbacks
|
||||
|
||||
**Incoming:**
|
||||
- `POST /webhooks/{name}` - Named webhooks with HMAC-SHA256 verification (`src/automation/webhooks.ts`)
|
||||
- `POST /gmail/push` - Google Pub/Sub push notifications for Gmail (`src/automation/gmail.ts`)
|
||||
|
||||
**Outgoing:**
|
||||
- None (no outbound webhooks — all communication goes through channel adapters)
|
||||
|
||||
---
|
||||
|
||||
*Integration audit: 2026-02-09*
|
||||
Reference in New Issue
Block a user