From f4a61acf7fa4be2e68d2483e58f6dc9373764f0d Mon Sep 17 00:00:00 2001 From: William Valentin Date: Wed, 13 May 2026 08:58:25 -0700 Subject: [PATCH] docs: add n8n implementation handoff --- .../Automation/n8n Implementation Handoff.md | 349 ++++++++++++++++++ 1 file changed, 349 insertions(+) create mode 100644 swarm-common/obsidian-vault/will/will-shared-zap/Infrastructure/Automation/n8n Implementation Handoff.md diff --git a/swarm-common/obsidian-vault/will/will-shared-zap/Infrastructure/Automation/n8n Implementation Handoff.md b/swarm-common/obsidian-vault/will/will-shared-zap/Infrastructure/Automation/n8n Implementation Handoff.md new file mode 100644 index 0000000..6a082c6 --- /dev/null +++ b/swarm-common/obsidian-vault/will/will-shared-zap/Infrastructure/Automation/n8n Implementation Handoff.md @@ -0,0 +1,349 @@ +--- +title: n8n Implementation Handoff +area: automation +tags: [automation, n8n, handoff, atlas] +created: 2026-05-13 +updated: 2026-05-13 +status: active +related: [[Infrastructure/Automation/n8n Workflows]], [[Infrastructure/Automation/n8n IMAP Triage Pipeline]], [[Infrastructure/Automation/n8n Nightly Vault Sync]], [[Infrastructure/Services/Docker Services]] +--- + +# n8n Implementation Handoff + +## Current state + +n8n is running on `n8n-agent` at `http://127.0.0.1:18808`. + +Last verified on 2026-05-13: + +- Container: `n8n-agent` running and healthy. +- Health endpoint: `GET /healthz` returns `{"status":"ok"}`. +- Workflow export: `n8n export:workflow --all` succeeds. +- Active workflows: 11. +- Archived workflows: 2 unrecoverable duplicate IMAP workflows archived after SQLite recovery. + +## Implemented and active + +### Swarm Health Watchdog + +- Workflow ID: `lDKocSFXBQWQrDd3` +- Status: active +- Trigger: every 15 minutes +- Checks: + - Brave MCP `18802` + - SearXNG `18803` + - LiteLLM `18804` + - Kokoro `18805` + - llama.cpp `18806` + - Ollama embeddings `18807` + - n8n `18808` + - Whisper `18811` +- Current behavior: + - Runs from inside the n8n container. + - Uses Docker bridge gateway `172.19.0.1` for host-published services. + - Uses `127.0.0.1:5678` for n8n self-health. + - Alerts Telegram and Discord after two consecutive failures. + - Sends recovery messages when a previously-alerted service becomes healthy again. +- Remaining improvement: + - Add Docker health-state enrichment. This requires either mounting Docker socket read-only or adding a host-side helper endpoint/script. Do not mount the Docker socket casually; decide the safer boundary first. + +### n8n Failure Digest + +- Workflow ID: `G9ylNbHbnJ6fWX2C` +- Status: active +- Trigger: hourly plus manual trigger +- Current behavior: + - Queries recent failed n8n executions. + - Groups failures by workflow/node/error. + - Sends a concise Telegram digest. +- Remaining improvement: + - Add Discord delivery to ops channel. + - Tune grouping/windowing after observing real failure volume. + - Add first-seen/last-seen persistence if the current digest is too repetitive. + +### Local AI Inbox Triage / IMAP + +- Primary workflow ID: `9sFwRyUDz51csAp7` +- Status: active +- Current behavior: + - Polls IMAP every 15 minutes. + - Uses local LLM classification. + - Sends signal digests to Telegram. + - Writes per-email notes to Obsidian. +- Archived duplicates: + - `xjUoQf97TkBrawc8` — `IMAP Inbox Triage + Obsidian Notes (squareffect)` + - `kHDK9QdUSiAJ8rCM` — `IMAP Inbox Triage + Obsidian Notes (wills-portal)` +- Remaining improvement: + - Rebuild any needed second-mailbox triage flows from scratch instead of unarchiving the corrupted duplicates. + - Add draft-reply generation for reply-needed messages. + - Add clearer classes: `needs_reply`, `FYI`, `receipt`, `newsletter`, `noise`, `action_required`. + - Add explicit Obsidian destinations per class. + +### Gmail Inbox Monitor + Obsidian Notes + +- Workflow ID: `whtdorf7yJMVYeHm` +- Status: active with warning +- Current issue: + - n8n startup logs still show a Gmail Trigger warning: `undefined`. +- Remaining implementation: + - Inspect the Gmail Trigger node and credential state in the n8n UI/API. + - Confirm whether the trigger is actually receiving new mail. + - If the trigger is flaky, replace it with scheduled Gmail API polling or IMAP polling. + - Add the same triage classes and draft-reply behavior as the IMAP flow. + +### Calendar to Obsidian Notes + +- Workflow ID: `QRCCdHNXZUHc2Oz4` +- Status: active +- Current behavior: + - Fetches upcoming calendar events. + - Writes or updates notes in the shared vault. +- Remaining improvement: + - Add meeting-prep note template. + - Add after-meeting follow-up path that collects event + notes + transcript and produces action items. + +### Nightly Obsidian Vault Sync + +- Workflow ID: `75JCevkdgkyCr2qH` +- Status: active +- Trigger: nightly +- Current behavior: + - Generates nightly operational note using local LLM and live service context. +- Remaining improvement: + - Align with the desired evening digest shape: jobs run today, failed jobs, reminders, short summaries. + - Add n8n Failure Digest summary as an input. + +### Daily OpenClaw Session Digest + +- Workflow ID: `qqYwAD05AvRHrHPc` +- Status: active +- Trigger: nightly +- Current behavior: + - Summarizes recent OpenClaw sessions into Obsidian. +- Remaining improvement: + - Decide whether to rename/reframe this as a broader Atlas/Hermes daily agent digest. + - Pull in Hermes cron/gateway/Kanban activity once useful. + +### Voice Memo Capture + +- Workflow ID: `El1BHJZ56JlzhrRZ` +- Status: active +- Type: webhook +- Current behavior: + - Accepts an audio URL. + - Downloads audio. + - Transcribes with local Whisper on `18811`. + - Summarizes with local llama.cpp. + - Writes transcript/summary/action items to Obsidian. + - Sends a Telegram notification. +- Remaining improvement: + - Add native Telegram/Discord voice-message ingress instead of requiring an audio URL. + - Add optional Kokoro read-back of summary. + - Add durable action-item routing to notes/task queue. + +### Web-to-Notes Capture + +- Workflow ID: `GSmzuA5dgGgyRg5v` +- Status: active +- Type: webhook +- Current behavior: + - Accepts a URL. + - Fetches the page. + - Extracts readable text. + - Summarizes with local llama.cpp. + - Writes markdown to Obsidian. +- Remaining improvement: + - Add YouTube transcript handling. + - Add PDF handling. + - Add claim extraction and source metadata. + - Add optional Atlas/Hermes higher-quality synthesis for important captures. + +### OpenClaw Action Bus / Reminder Webhook + +- Action Bus ID: `Jwi54VWMdlLqYnRo` +- Reminder Webhook ID: `RUR1CGn0ikkxbPin` +- Status: active +- Current behavior: + - Provides stable webhook ingress for approved automation and reminders. +- Remaining improvement: + - Define a clean Atlas-facing webhook catalog with paths, auth, payload schema, and response shape. + - Add handlers for: + - `process_url` + - `summarize_pdf` + - `add_reminder` + - `sync_vault` + - `run_health_check` + - `process_voice_memo` + +## Not yet implemented + +### Morning brief + +Desired scope: + +- Calendar for today. +- Email highlights / needs-reply items. +- Weather for Seattle. +- Tasks/reminders. +- Relevant news. +- Swarm/n8n health rollup. + +Recommended implementation: + +1. Build n8n scheduled workflow for data collection and retries. +2. Use local LLM for cheap classification/short summarization. +3. Call Atlas/Hermes only for final synthesis if quality matters. +4. Deliver to Telegram/Discord ops/home channel. +5. Save brief to Obsidian daily note. + +### Evening digest + +Desired scope: + +- What ran today. +- Failed n8n executions. +- Failed cron jobs/timers. +- Reminders due soon. +- New notes/captures. +- Short summaries and next actions. + +Recommended implementation: + +1. Extend Nightly Obsidian Vault Sync or create a separate evening digest workflow. +2. Pull n8n Failure Digest output, cron/timer health, and Obsidian note deltas. +3. Send concise report to Telegram/Discord. +4. Save durable digest note in Obsidian. + +### Weekly review + +Desired scope: + +- Obsidian note changes. +- Calendar summary. +- GitHub/Linear activity. +- Inbox triage rollup. +- Open tasks and decisions. + +Recommended implementation: + +1. Use n8n weekly trigger. +2. Collect structured inputs from calendar, Obsidian, GitHub/Linear, email summaries. +3. Use Atlas/Hermes or cloud model for final synthesis. +4. Write `Notes/YYYY-MM-DD Weekly Review.md`. + +### Obsidian Semantic Index + +Desired scope: + +- Watch vault changes. +- Chunk changed notes. +- Embed with Ollama on `18807` using `nomic-embed-text`. +- Store vectors locally. +- Enable semantic search / RAG for Atlas. + +Recommended implementation options: + +1. Prefer Hermes `rag-search`/local ChromaDB if already available and stable. +2. If n8n owns the trigger, have n8n call a local indexing webhook/script rather than implementing vector DB logic entirely in n8n. +3. Use file-change polling if native file watch is unreliable in Docker/virtiofs. + +Open questions: + +- Which vector store should be canonical: Hermes rag-search ChromaDB, a separate Chroma instance, SQLite vector extension, or another local store? +- Should n8n trigger indexing, or should Atlas/Hermes own indexing and n8n only notify? + +### Personal data routing + +Desired scope: + +- Gmail/IMAP items to Obsidian. +- Telegram/Discord messages to notes or task queues. +- Calendar events to prep docs. +- YouTube/RSS/blog links to summaries. + +Current status: + +- Email and calendar pieces exist. +- URL capture exists. +- Telegram/Discord message-to-note routing is not fully built. +- RSS/blog monitoring is not fully built in n8n. + +Recommended implementation: + +1. Define routing taxonomy and destination folders/tags. +2. Implement one source at a time. +3. Avoid dumping noisy raw messages into Obsidian; use classification gates. + +## Recommended next order of work + +1. Fix Gmail Trigger warning. + - Goal: no startup warning and confirmed Gmail ingestion. + - Fallback: replace trigger with scheduled polling. + +2. Add Docker health-state enrichment to Swarm Health Watchdog. + - Prefer safe host-side helper over raw Docker socket mount. + - Include last Docker health state and suggested fix in alerts. + +3. Build Morning Brief v1. + - Start small: calendar + weather + email highlights + swarm/n8n health. + - Deliver to Telegram/Discord and save to Obsidian. + +4. Extend Evening Digest. + - Use n8n Failure Digest plus workflow/timer run summary. + - Include failed jobs and next recommended action. + +5. Implement Obsidian Semantic Index. + - Decide canonical vector store first. + - Use Ollama embeddings on `18807`. + - Add incremental update path. + +6. Upgrade Web-to-Notes Capture. + - Add PDF and YouTube transcript support. + - Add source metadata and claim extraction. + +7. Upgrade Voice Memo Pipeline. + - Add native Telegram/Discord voice ingestion. + - Add optional Kokoro audio summary. + +8. Define webhook action bus catalog. + - Document stable endpoints and schemas. + - Add `process_url`, `summarize_pdf`, `add_reminder`, `sync_vault`, `run_health_check`. + +## Verification commands + +Run from host: + +```bash +cd /home/will/lab/swarm +make status +curl -fsS --max-time 3 http://127.0.0.1:18808/healthz + +docker inspect --format '{{.Name}} {{.State.Status}} {{if .State.Health}}{{.State.Health.Status}}{{else}}no-health{{end}} restarts={{.RestartCount}}' n8n-agent + +docker exec n8n-agent n8n export:workflow --all --output=/tmp/workflows-verify.json +``` + +Inspect workflow inventory from SQLite copy: + +```bash +tmp=/tmp/n8n-check-$(date +%s) +mkdir -p "$tmp" +docker cp n8n-agent:/home/node/.n8n/database.sqlite "$tmp/database.sqlite" +sqlite3 "$tmp/database.sqlite" "select name, active, isArchived from workflow_entity order by name;" +``` + +Check logs without printing secrets: + +```bash +docker logs n8n-agent --tail 120 +``` + +## Pitfalls + +- Do not assume `/healthz` means all workflows are valid. Verify with `n8n export:workflow --all`. +- If SQLite corruption returns, copy DB files first, then recover offline. Do not make blind in-place edits. +- Validate `workflow_entity` JSON-ish fields after SQLite recovery: `nodes`, `connections`, `settings`, `staticData`, `pinData`, `meta`. +- Empty strings in these fields can crash n8n even when SQLite integrity is `ok`. +- From inside n8n, host-published swarm services use Docker bridge gateway `172.19.0.1:`. +- From inside n8n, n8n self-calls should use `127.0.0.1:5678`, not host port `18808`. +- Do not commit DB backups, workflow execution history, secrets, or runtime state.