diff --git a/CHANGELOG.md b/CHANGELOG.md index 1b7cfa4..d176b31 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,23 @@ All notable changes to Flynn are documented in this file. ### Added +- **Inbound Webhooks** -- HTTP endpoints (`POST /webhooks/:name`) that trigger agent + processing. Config-driven with per-webhook HMAC signature verification, message + template rendering (`{{body}}`, `{{json.field}}`), and output channel routing. + Bypasses gateway token auth in favour of per-webhook secrets. +- **Heartbeat Monitor** -- Periodic health check system with 5 configurable checks: + gateway responsiveness, model router status, channel adapter connectivity, memory + store accessibility, and disk space. Sends failure notifications after a configurable + threshold and recovery notifications when health restores. +- **Vector Memory Search** -- Hybrid keyword + semantic search for the memory system. + Provider-agnostic embeddings (OpenAI, Gemini, Ollama, llama.cpp) stored in SQLite + with brute-force cosine similarity. Background indexer processes dirty namespaces + every 30s. Configurable hybrid weighting (0.0 = keyword only, 1.0 = vector only). + Graceful fallback to keyword search when embeddings are unavailable. +- **Docker Deployment** -- Multi-stage Dockerfile (node:22-alpine build + runtime), + `.dockerignore`, and `docker-compose.yml`. Native dependency handling for + `better-sqlite3`. `FLYNN_DATA_DIR` environment variable for configurable data + directory in container deployments. - **Agent Tools: sessions.\*** -- 4 new agent-callable tools (`sessions.list`, `sessions.history`, `sessions.create`, `sessions.delete`) wrapping SessionManager for runtime session management by the AI agent diff --git a/README.md b/README.md index 269bfee..b600cef 100644 --- a/README.md +++ b/README.md @@ -19,6 +19,10 @@ Self-hosted personal AI assistant with Telegram and Terminal interfaces. - **Session Transfer**: Move conversations between frontends - **CLI**: Full command-line interface (`flynn start`, `send`, `doctor`, etc.) - **Cron Scheduling**: Automated messages on cron schedules with output routing +- **Inbound Webhooks**: HTTP endpoints that trigger agent processing with HMAC auth and template rendering +- **Heartbeat Monitor**: Periodic health checks (gateway, model, channels, memory, disk) with failure notifications +- **Vector Memory Search**: Hybrid keyword + semantic search with embeddings (OpenAI, Gemini, Ollama, llama.cpp) +- **Docker Deployment**: Multi-stage Dockerfile and docker-compose.yml for production containers - **Health Diagnostics**: `flynn doctor` validates config, connectivity, and system state - **MCP Integration**: External tool servers via Model Context Protocol - **Skills System**: Extensible capability packages (bundled, managed, workspace tiers) @@ -253,6 +257,184 @@ automation: | `timezone` | no | IANA timezone (defaults to system timezone) | | `enabled` | no | Whether the job is active (default: `true`) | +## Inbound Webhooks + +HTTP endpoints that trigger agent processing. Each webhook accepts POST requests, optionally verifies an HMAC signature, renders a message template, and routes the agent's response to an output channel. + +```yaml +automation: + webhooks: + - name: github-push + secret: "whsec_..." # HMAC secret for signature verification + message: "GitHub push to {{json.repository.full_name}}: {{json.head_commit.message}}" + output: + channel: telegram + peer: "123456789" + + - name: alertmanager + message: "Alert: {{json.alerts.0.annotations.summary}}" + output: + channel: discord + peer: "channel-id" +``` + +Webhooks are available at `POST /webhooks/:name` on the gateway HTTP server. They bypass gateway token auth and use their own per-webhook HMAC verification instead. + +### Webhook Config Fields + +| Field | Required | Description | +|-------|----------|-------------| +| `name` | yes | Unique webhook identifier (used in URL path) | +| `secret` | no | HMAC secret for `X-Webhook-Signature` header verification (SHA-256) | +| `message` | no | Template for the message sent to the agent (default: `{{body}}`) | +| `output.channel` | yes | Channel name to route the response (e.g. `telegram`) | +| `output.peer` | yes | Peer/chat ID on the output channel | +| `enabled` | no | Whether the webhook is active (default: `true`) | + +### Template Variables + +| Variable | Description | +|----------|-------------| +| `{{body}}` | Raw request body as string | +| `{{json.field}}` | Extract a field from JSON body (dot notation for nested fields) | + +## Heartbeat Monitor + +Periodic health checks that validate system components and notify a configured channel on failure. + +```yaml +automation: + heartbeat: + enabled: true + interval: "5m" # Check every 5 minutes + checks: [gateway, model, channels, memory, disk] + notify: + channel: telegram + peer: "123456789" + failure_threshold: 2 # Notify after 2 consecutive failures + disk_threshold_mb: 100 # Warn when <100MB free +``` + +### Heartbeat Checks + +| Check | What it validates | +|-------|-------------------| +| `gateway` | HTTP server is responsive | +| `model` | Model router has an active tier configured | +| `channels` | At least one channel adapter is connected | +| `memory` | Memory directory is readable and writable | +| `disk` | Free disk space exceeds threshold | + +The monitor sends a notification when failures reach the configured threshold and a recovery notification when all checks pass again. + +### Heartbeat Config Fields + +| Field | Required | Description | +|-------|----------|-------------| +| `enabled` | no | Enable the heartbeat monitor (default: `false`) | +| `interval` | no | Check interval: `60s`, `5m`, `1h` (default: `5m`) | +| `checks` | no | Which checks to run (default: all five) | +| `notify.channel` | no | Channel to send failure/recovery notifications | +| `notify.peer` | no | Peer/chat ID for notifications | +| `failure_threshold` | no | Consecutive failures before notifying (default: `2`) | +| `disk_threshold_mb` | no | Disk space warning threshold in MB (default: `100`) | + +## Vector Memory Search + +The memory system supports hybrid search combining keyword matching with semantic vector similarity. When embeddings are enabled, `memory.search` uses both approaches and merges results with configurable weighting. + +```yaml +memory: + enabled: true + auto_extract: true + max_context_tokens: 2000 + embedding: + enabled: true + provider: openai # openai, gemini, ollama, llamacpp + model: text-embedding-3-small + api_key: "${OPENAI_API_KEY}" + chunk_size: 512 # Tokens per chunk + chunk_overlap: 50 # Overlap between chunks + top_k: 5 # Top results from vector search + hybrid_weight: 0.7 # 0.0 = keyword only, 1.0 = vector only +``` + +### Embedding Providers + +| Provider | Config | +|----------|--------| +| OpenAI | `provider: openai`, `api_key`, `model` (default: `text-embedding-3-small`) | +| Gemini | `provider: gemini`, `api_key`, `model` | +| Ollama | `provider: ollama`, `endpoint` (default: localhost:11434), `model` | +| llama.cpp | `provider: llamacpp`, `endpoint`, optional `model` | + +Embeddings are indexed in the background — when memory is written, the namespace is marked dirty and re-indexed within 30 seconds. The vector index is stored in `vectors.db` alongside the session database. + +When embeddings are disabled or the provider is unreachable, search falls back gracefully to keyword matching. + +### Embedding Config Fields + +| Field | Required | Description | +|-------|----------|-------------| +| `enabled` | no | Enable vector search (default: `false`) | +| `provider` | no | Embedding provider (default: `openai`) | +| `model` | no | Embedding model name (default: `text-embedding-3-small`) | +| `endpoint` | no | Provider endpoint (required for `ollama`/`llamacpp`) | +| `api_key` | no | API key (required for `openai`/`gemini`) | +| `dimensions` | no | Vector dimensions (auto-detected from model if not set) | +| `chunk_size` | no | Max tokens per chunk (default: `512`) | +| `chunk_overlap` | no | Token overlap between chunks (default: `50`) | +| `top_k` | no | Number of vector results to return (default: `5`) | +| `hybrid_weight` | no | Vector vs keyword weight, 0.0-1.0 (default: `0.7`) | + +## Docker Deployment + +Flynn includes a production-ready Dockerfile with multi-stage build. + +```bash +# Build the image +docker build -t flynn . + +# Run with config and data volumes +docker run -d \ + --name flynn \ + -p 18800:18800 \ + -v ./config.yaml:/config/config.yaml:ro \ + -v flynn-data:/data \ + -e ANTHROPIC_API_KEY=sk-ant-... \ + flynn +``` + +Or use the included `docker-compose.yml`: + +```bash +# Copy your config +cp ~/.config/flynn/config.yaml ./config.yaml + +# Start with compose +docker compose up -d + +# View logs +docker compose logs -f +``` + +### Docker Environment Variables + +| Variable | Description | +|----------|-------------| +| `FLYNN_CONFIG` | Config file path (default: `/config/config.yaml`) | +| `FLYNN_DATA_DIR` | Data directory path (default: `/data`) | +| `ANTHROPIC_API_KEY` | Anthropic API key | +| `OPENAI_API_KEY` | OpenAI API key | +| `TELEGRAM_BOT_TOKEN` | Telegram bot token | + +### Volumes + +| Mount Point | Purpose | +|-------------|---------| +| `/config/config.yaml` | Configuration file (read-only) | +| `/data` | Persistent data (sessions DB, memory files, vector index) | + ## Doctor Diagnostics `flynn doctor` runs 10 health checks to validate your setup: @@ -323,14 +505,14 @@ src/ │ └── lib/ # WebSocket RPC client ├── hooks/ # Confirmation engine ├── mcp/ # MCP tool server integration -├── memory/ # Persistent memory store +├── memory/ # Persistent memory store + vector search ├── models/ # Model providers + router + media pipeline ├── prompt/ # System prompt templating ├── sandbox/ # Docker sandboxing ├── session/ # SQLite persistence ├── skills/ # Skill packages ├── tools/ # Builtin tools (shell, file, web, browser, process, media) -└── automation/ # Cron scheduler +└── automation/ # Cron scheduler, webhooks, heartbeat monitor ``` ## Development @@ -354,7 +536,8 @@ pnpm test | Variable | Description | |----------|-------------| -| `FLYNN_CONFIG` | Override config path | +| `FLYNN_CONFIG` | Override config file path | +| `FLYNN_DATA_DIR` | Override data directory (default: `~/.local/share/flynn`) | | `ANTHROPIC_API_KEY` | Anthropic API key (fallback) | | `OPENAI_API_KEY` | OpenAI API key (fallback) |