will/flynn
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
671ec035e90a5232819888cfcae17045557e89a0
flynn/.planning/codebase/INTEGRATIONS.md
T

13 KiB
Raw Blame History

External Integrations

Analysis Date: 2026-02-09

AI Model Providers

Flynn supports 10 model providers via a unified ModelClient interface (src/models/types.ts). Each provider implements chat() and optionally chatStream(). The ModelRouter (src/models/router.ts) manages tier-based routing (fast/default/complex/local) with fallback chains.

Anthropic:

  • SDK: @anthropic-ai/sdk (src/models/anthropic.ts)
  • Auth: ANTHROPIC_API_KEY env var or api_key in config
  • Features: Streaming, tool use, extended thinking mode, multimodal (images)
  • Extended thinking: { type: 'enabled', budget_tokens: 4096 } on request

OpenAI:

  • SDK: openai (src/models/openai.ts)
  • Auth: OPENAI_API_KEY env var or api_key in config
  • Features: Tool use, multimodal (images via data URIs or URLs)
  • Also powers: OpenRouter, ZhipuAI, xAI via baseURL override

Google Gemini:

  • SDK: @google/generative-ai (src/models/gemini.ts)
  • Auth: GOOGLE_API_KEY env var or api_key in config
  • Features: Streaming, tool use, extended thinking, multimodal

AWS Bedrock:

  • SDK: @aws-sdk/client-bedrock-runtime (src/models/bedrock.ts)
  • Auth: AWS_REGION env var + IAM credentials or explicit accessKeyId/secretAccessKey in config
  • Features: Streaming (ConverseStream), tool use, multimodal
  • Models: Meta Llama, Amazon Titan (cost-tracked in src/models/costs.ts)

GitHub Models (Copilot):

  • SDK: openai (OpenAI-compatible API) (src/models/github.ts)
  • Auth: GITHUB_TOKEN env var or OAuth device flow (src/auth/github.ts)
  • Endpoint: https://api.githubcopilot.com
  • Auto-fallback: When an Anthropic tier fails, Flynn automatically tries the same model via GitHub Models before the global fallback chain (src/daemon/index.ts createAutoFallbackClient())
  • OAuth device flow: Uses client ID Ov23li8tweQw6odWQebz, stores token at ~/.config/flynn/auth.json

OpenRouter:

  • SDK: openai with baseURL: https://openrouter.ai/api/v1 (src/daemon/index.ts)
  • Auth: OPENROUTER_API_KEY env var or api_key in config

ZhipuAI:

  • SDK: openai with baseURL: https://api.z.ai/api/paas/v4 (src/daemon/index.ts)
  • Auth: ZHIPUAI_API_KEY env var or api_key in config

xAI (Grok):

  • SDK: openai with baseURL: https://api.x.ai/v1 (src/daemon/index.ts)
  • Auth: XAI_API_KEY env var or api_key in config

Ollama (Local):

  • SDK: ollama (src/models/local/ollama.ts)
  • Auth: None (local server)
  • Endpoint: Configurable host (default: http://localhost:11434)
  • Config: num_gpu option for GPU layer control

llama.cpp (Local):

  • SDK: Raw fetch HTTP calls (src/models/local/llamacpp.ts)
  • Auth: Optional auth_token header
  • Endpoint: Configurable (default: http://localhost:8080)

Embedding Providers

Embedding providers (src/memory/embeddings.ts) power the hybrid vector + keyword search system. Factory function: createEmbeddingProvider().

OpenAI Embeddings:

  • SDK: openai (lazy import)
  • Auth: OPENAI_API_KEY or config api_key
  • Default model: text-embedding-3-small, default dims: 1536

Gemini Embeddings:

  • SDK: @google/generative-ai (lazy import)
  • Auth: GOOGLE_API_KEY or config api_key
  • Uses batchEmbedContents for efficiency, default dims: 768

Ollama Embeddings:

  • SDK: ollama (lazy import)
  • Auth: None (local)
  • Configurable host endpoint, default dims: 768

LlamaCpp Embeddings:

  • SDK: Raw fetch to /embedding endpoint
  • Auth: None
  • Default endpoint: http://localhost:8080, default dims: 768

Voyage AI Embeddings:

  • SDK: openai (OpenAI-compatible API, lazy import)
  • Auth: VOYAGE_API_KEY env var or config api_key
  • Endpoint: https://api.voyageai.com/v1, default dims: 1024

Data Storage

Session Database (SQLite):

  • Library: better-sqlite3 (src/session/store.ts)
  • Location: {dataDir}/sessions.db
  • Schema: messages table with id, session_id, role, content, created_at
  • TTL-based pruning: Configurable via sessions.ttl (default: 30 days), hourly cleanup

Vector Database (SQLite):

  • Library: better-sqlite3 (src/memory/vector-store.ts)
  • Location: {dataDir}/vectors.db
  • Stores embedding chunks as Float32Array BLOBs
  • Content hashing for deduplication
  • Background indexer runs every 30 seconds

Memory Store (Filesystem):

  • Location: {dataDir}/memory/ (src/memory/store.ts)
  • Format: Markdown files organized by namespace
  • Layout: global.md, user.md, sessions/{id}.md
  • Hybrid search: Keyword + vector (configurable weight via hybrid_weight, default 0.7)

File Storage:

  • Local filesystem only — no cloud object storage

Caching:

  • In-memory response cache for web fetch tool (5-minute TTL) (src/tools/builtin/web-fetch.ts)
  • No external cache service (Redis, etc.)

Channel Adapters (Messaging Platforms)

All adapters implement ChannelAdapter interface (src/channels/types.ts): connect(), disconnect(), send(), onMessage().

Telegram:

  • SDK: grammy (src/channels/telegram/)
  • Auth: Bot token via telegram.bot_token config
  • Features: Long polling, chat ID allowlist, mention requirement, pairing codes, image/audio attachments

Discord:

  • SDK: discord.js (src/channels/discord/)
  • Auth: Bot token via discord.bot_token config
  • Features: Guild/channel allowlists, mention requirement, pairing codes

Slack:

  • SDK: @slack/bolt (src/channels/slack/)
  • Auth: bot_token, app_token, signing_secret in config
  • Features: Socket mode, channel allowlists, mention requirement, pairing codes

WhatsApp:

  • SDK: whatsapp-web.js (src/channels/whatsapp/)
  • Auth: QR code scanning (web client emulation)
  • Features: Number/group allowlists, mention requirement, custom data directory, pairing codes

WebChat:

  • Implementation: Gateway WebSocket bridge (src/channels/webchat/)
  • Auth: Gateway token or Tailscale identity
  • UI: Vanilla JS dashboard at src/gateway/ui/ (HTML + CSS + JS, no framework)

Authentication & Identity

GitHub OAuth (Device Flow):

  • Implementation: src/auth/github.ts
  • Client ID: Ov23li8tweQw6odWQebz (GitHub Copilot)
  • Flow: Device code → User authorization → Token polling
  • Storage: ~/.config/flynn/auth.json (600 permissions)
  • Priority: GITHUB_TOKEN env → stored OAuth token → null

Gateway Auth:

  • Static bearer token (server.token in config)
  • Tailscale identity header trust (server.tailscale_identity)
  • HTTP auth optional (server.auth_http)
  • Gateway lock: Single-client WebSocket mode (server.lock)

DM Pairing Codes:

  • Implementation: src/channels/pairing.ts, src/session/store.ts (SQLite persistence)
  • Purpose: Authenticate unknown senders via one-time codes
  • Config: pairing.enabled, pairing.code_ttl (default 5m), pairing.code_length (default 6)
  • Gateway handlers for code generation/verification
  • TUI /pair command execution (generate/list/revoke) in src/frontends/tui/minimal.ts
  • Persistence: PairingStore interface with SQLite pairing_approved table -- approved senders survive daemon restarts

Gmail OAuth2:

  • SDK: googleapis (src/automation/gmail.ts)
  • Credentials: ~/.config/flynn/gmail-credentials.json
  • Token: ~/.config/flynn/gmail-token.json
  • Setup: flynn gmail-auth CLI command

Automation

Cron Scheduler:

  • Library: croner (src/automation/cron.ts)
  • Config: automation.cron[] — each job has name, schedule, message, output.channel, output.peer
  • Implements ChannelAdapter to inject cron-triggered messages into the channel registry
  • Features: Enable/disable per job, timezone support, runtime management tools

Webhooks:

  • Implementation: src/automation/webhooks.ts
  • Auth: HMAC-SHA256 signature verification (X-Webhook-Signature header)
  • Templates: {{body}} and {{json.field}} placeholders
  • Route: POST /webhooks/{name} on the gateway HTTP server
  • Config: automation.webhooks[] with name, secret, message, output

Gmail Watcher:

  • SDK: googleapis (src/automation/gmail.ts)
  • Modes: Pub/Sub push notifications or polling fallback
  • Pub/Sub topic: projects/flynn-agent/topics/gmail-push
  • Watch renewal: Every 6 days (Google watch expires at ~7 days)
  • Config: automation.gmail with watch_labels, poll_interval, history_start
  • Route: POST /gmail/push on gateway for Pub/Sub push

Heartbeat Monitor:

  • Implementation: src/automation/heartbeat.ts
  • Checks: gateway, model, channels, memory, disk
  • Config: automation.heartbeat with interval, checks, failure_threshold, disk_threshold_mb
  • Notification: Sends to configured channel/peer on failures

Web & Content Tools

Web Search (Brave / SearXNG):

  • Implementation: src/tools/builtin/web-search.ts
  • Brave Search API: https://api.search.brave.com/res/v1/web/search
    • Auth: X-Subscription-Token header via web_search.api_key
  • SearXNG: Self-hosted instance via web_search.endpoint
    • Auth: None (private instance)
  • Config: web_search.provider (brave or searxng), web_search.max_results

Web Fetch (Readability):

  • Libraries: linkedom, @mozilla/readability, turndown (src/tools/builtin/web-fetch.ts)
  • Features: HTML → Markdown conversion, article extraction, response caching (5min TTL)
  • Truncation: 50,000 character max

Browser Automation:

  • Library: puppeteer-core (src/tools/builtin/browser/)
  • Config: browser.executable_path or browser.ws_endpoint
  • Features: Headless browsing, page management, screenshots
  • Limits: browser.max_pages (default 5), browser.default_timeout (default 30s)

Audio Transcription

Whisper-Compatible API:

  • Implementation: src/models/media.ts
  • Endpoint: Configurable via audio.transcription_endpoint
  • Auth: audio.transcription_api_key (Bearer token)
  • Model: audio.transcription_model (default: whisper-1)
  • Supported formats: OGG, MP3, WAV, WebM, MP4, M4A
  • Integration: Auto-transcribes audio attachments from channels before model processing

MCP (Model Context Protocol)

MCP Client:

  • SDK: @modelcontextprotocol/sdk (src/mcp/client.ts)
  • Transport: stdio (spawns external processes)
  • Config: mcp.servers[] with name, command, args, env, cwd
  • Bridge: MCP tools auto-registered in Flynn's tool registry (src/mcp/bridge.ts)
  • Management: McpManager starts/stops all configured servers (src/mcp/manager.ts)

Docker Sandbox

Per-Session Containers:

  • Implementation: src/sandbox/manager.ts, src/sandbox/docker.ts
  • Config: sandbox.image (default: node:22-slim), sandbox.network (default: none), sandbox.memory_limit, sandbox.cpu_limit
  • Features: Lazily created per session, replaces shell.exec and process.start tools with sandboxed versions
  • Prerequisite: Docker daemon available

Networking & Exposure

Gateway Server:

  • Protocol: WebSocket (JSON-RPC) + HTTP (src/gateway/server.ts)
  • Default port: 18800
  • Binding: 127.0.0.1 (localhost only) or 0.0.0.0
  • Features: LaneQueue for request ordering, session bridge, static file serving for dashboard

Tailscale Serve:

  • Implementation: src/gateway/tailscale.ts
  • Purpose: Expose gateway HTTPS endpoint on tailnet
  • Config: server.tailscale.serve, server.tailscale.hostname, server.tailscale.port
  • Prerequisite: Tailscale CLI installed and daemon running

Monitoring & Observability

Error Tracking:

  • None (console.error only)

Logging:

  • console.log / console.error / console.debug throughout
  • No structured logging framework

Cost Tracking:

  • Built-in: src/models/costs.ts with per-million-token pricing for known models
  • Tracks: Anthropic, OpenAI, Gemini, xAI, Bedrock models
  • GitHub Copilot models tracked at $0 (subscription-included)
  • Usage exposed via /usage command and gateway system.usage RPC

CI/CD & Deployment

Hosting:

  • Self-hosted (designed for personal deployment)
  • Process supervisor expected for restarts (exit code 75 = restart signal)

CI Pipeline:

  • Not detected in repository

Environment Configuration

Required env vars (minimum viable):

  • ANTHROPIC_API_KEY (or other model provider key)
  • FLYNN_TELEGRAM_TOKEN (if using default Telegram channel)

Optional env vars (by feature):

  • OPENAI_API_KEY - OpenAI models and embeddings
  • GOOGLE_API_KEY - Gemini models and embeddings
  • GITHUB_TOKEN - GitHub Models / Copilot access
  • AWS_REGION - Bedrock region
  • OPENROUTER_API_KEY - OpenRouter access
  • ZHIPUAI_API_KEY - ZhipuAI access
  • XAI_API_KEY - xAI (Grok) access
  • VOYAGE_API_KEY - Voyage AI embeddings
  • FLYNN_DATA_DIR - Custom data directory

Secrets location:

  • API keys: YAML config (with ${ENV_VAR} expansion) or environment variables
  • OAuth tokens: ~/.config/flynn/auth.json (GitHub), ~/.config/flynn/gmail-token.json (Gmail)
  • .env.example present at project root

Webhooks & Callbacks

Incoming:

  • POST /webhooks/{name} - Named webhooks with HMAC-SHA256 verification (src/automation/webhooks.ts)
  • POST /gmail/push - Google Pub/Sub push notifications for Gmail (src/automation/gmail.ts)

Outgoing:

  • None (no outbound webhooks — all communication goes through channel adapters)

Integration audit: 2026-02-09

Reference in New Issue View Git Blame Copy Permalink