will/swarm-master
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add n8n implementation handoff

Browse Source
This commit is contained in:
William Valentin
2026-05-13 08:58:25 -07:00
parent 84043aebb8
commit f4a61acf7f
1 changed files with 349 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
swarm-common/obsidian-vault/will/will-shared-zap/Infrastructure/Automation/n8n Implementation Handoff.md
+349
View File
@@ -0,0 +1,349 @@
---
title: n8n Implementation Handoff
area: automation
tags: [automation, n8n, handoff, atlas]
created: 2026-05-13
updated: 2026-05-13
status: active
related: [[Infrastructure/Automation/n8n Workflows]], [[Infrastructure/Automation/n8n IMAP Triage Pipeline]], [[Infrastructure/Automation/n8n Nightly Vault Sync]], [[Infrastructure/Services/Docker Services]]
---
# n8n Implementation Handoff
## Current state
n8n is running on `n8n-agent` at `http://127.0.0.1:18808`.
Last verified on 2026-05-13:
- Container: `n8n-agent` running and healthy.
- Health endpoint: `GET /healthz` returns `{"status":"ok"}`.
- Workflow export: `n8n export:workflow --all` succeeds.
- Active workflows: 11.
- Archived workflows: 2 unrecoverable duplicate IMAP workflows archived after SQLite recovery.
## Implemented and active
### Swarm Health Watchdog
- Workflow ID: `lDKocSFXBQWQrDd3`
- Status: active
- Trigger: every 15 minutes
- Checks:
- Brave MCP `18802`
- SearXNG `18803`
- LiteLLM `18804`
- Kokoro `18805`
- llama.cpp `18806`
- Ollama embeddings `18807`
- n8n `18808`
- Whisper `18811`
- Current behavior:
- Runs from inside the n8n container.
- Uses Docker bridge gateway `172.19.0.1` for host-published services.
- Uses `127.0.0.1:5678` for n8n self-health.
- Alerts Telegram and Discord after two consecutive failures.
- Sends recovery messages when a previously-alerted service becomes healthy again.
- Remaining improvement:
- Add Docker health-state enrichment. This requires either mounting Docker socket read-only or adding a host-side helper endpoint/script. Do not mount the Docker socket casually; decide the safer boundary first.
### n8n Failure Digest
- Workflow ID: `G9ylNbHbnJ6fWX2C`
- Status: active
- Trigger: hourly plus manual trigger
- Current behavior:
- Queries recent failed n8n executions.
- Groups failures by workflow/node/error.
- Sends a concise Telegram digest.
- Remaining improvement:
- Add Discord delivery to ops channel.
- Tune grouping/windowing after observing real failure volume.
- Add first-seen/last-seen persistence if the current digest is too repetitive.
### Local AI Inbox Triage / IMAP
- Primary workflow ID: `9sFwRyUDz51csAp7`
- Status: active
- Current behavior:
- Polls IMAP every 15 minutes.
- Uses local LLM classification.
- Sends signal digests to Telegram.
- Writes per-email notes to Obsidian.
- Archived duplicates:
- `xjUoQf97TkBrawc8` — `IMAP Inbox Triage + Obsidian Notes (squareffect)`
- `kHDK9QdUSiAJ8rCM` — `IMAP Inbox Triage + Obsidian Notes (wills-portal)`
- Remaining improvement:
- Rebuild any needed second-mailbox triage flows from scratch instead of unarchiving the corrupted duplicates.
- Add draft-reply generation for reply-needed messages.
- Add clearer classes: `needs_reply`, `FYI`, `receipt`, `newsletter`, `noise`, `action_required`.
- Add explicit Obsidian destinations per class.
### Gmail Inbox Monitor + Obsidian Notes
- Workflow ID: `whtdorf7yJMVYeHm`
- Status: active with warning
- Current issue:
- n8n startup logs still show a Gmail Trigger warning: `undefined`.
- Remaining implementation:
- Inspect the Gmail Trigger node and credential state in the n8n UI/API.
- Confirm whether the trigger is actually receiving new mail.
- If the trigger is flaky, replace it with scheduled Gmail API polling or IMAP polling.
- Add the same triage classes and draft-reply behavior as the IMAP flow.
### Calendar to Obsidian Notes
- Workflow ID: `QRCCdHNXZUHc2Oz4`
- Status: active
- Current behavior:
- Fetches upcoming calendar events.
- Writes or updates notes in the shared vault.
- Remaining improvement:
- Add meeting-prep note template.
- Add after-meeting follow-up path that collects event + notes + transcript and produces action items.
### Nightly Obsidian Vault Sync
- Workflow ID: `75JCevkdgkyCr2qH`
- Status: active
- Trigger: nightly
- Current behavior:
- Generates nightly operational note using local LLM and live service context.
- Remaining improvement:
- Align with the desired evening digest shape: jobs run today, failed jobs, reminders, short summaries.
- Add n8n Failure Digest summary as an input.
### Daily OpenClaw Session Digest
- Workflow ID: `qqYwAD05AvRHrHPc`
- Status: active
- Trigger: nightly
- Current behavior:
- Summarizes recent OpenClaw sessions into Obsidian.
- Remaining improvement:
- Decide whether to rename/reframe this as a broader Atlas/Hermes daily agent digest.
- Pull in Hermes cron/gateway/Kanban activity once useful.
### Voice Memo Capture
- Workflow ID: `El1BHJZ56JlzhrRZ`
- Status: active
- Type: webhook
- Current behavior:
- Accepts an audio URL.
- Downloads audio.
- Transcribes with local Whisper on `18811`.
- Summarizes with local llama.cpp.
- Writes transcript/summary/action items to Obsidian.
- Sends a Telegram notification.
- Remaining improvement:
- Add native Telegram/Discord voice-message ingress instead of requiring an audio URL.
- Add optional Kokoro read-back of summary.
- Add durable action-item routing to notes/task queue.
### Web-to-Notes Capture
- Workflow ID: `GSmzuA5dgGgyRg5v`
- Status: active
- Type: webhook
- Current behavior:
- Accepts a URL.
- Fetches the page.
- Extracts readable text.
- Summarizes with local llama.cpp.
- Writes markdown to Obsidian.
- Remaining improvement:
- Add YouTube transcript handling.
- Add PDF handling.
- Add claim extraction and source metadata.
- Add optional Atlas/Hermes higher-quality synthesis for important captures.
### OpenClaw Action Bus / Reminder Webhook
- Action Bus ID: `Jwi54VWMdlLqYnRo`
- Reminder Webhook ID: `RUR1CGn0ikkxbPin`
- Status: active
- Current behavior:
- Provides stable webhook ingress for approved automation and reminders.
- Remaining improvement:
- Define a clean Atlas-facing webhook catalog with paths, auth, payload schema, and response shape.
- Add handlers for:
- `process_url`
- `summarize_pdf`
- `add_reminder`
- `sync_vault`
- `run_health_check`
- `process_voice_memo`
## Not yet implemented
### Morning brief
Desired scope:
- Calendar for today.
- Email highlights / needs-reply items.
- Weather for Seattle.
- Tasks/reminders.
- Relevant news.
- Swarm/n8n health rollup.
Recommended implementation:
1. Build n8n scheduled workflow for data collection and retries.
2. Use local LLM for cheap classification/short summarization.
3. Call Atlas/Hermes only for final synthesis if quality matters.
4. Deliver to Telegram/Discord ops/home channel.
5. Save brief to Obsidian daily note.
### Evening digest
Desired scope:
- What ran today.
- Failed n8n executions.
- Failed cron jobs/timers.
- Reminders due soon.
- New notes/captures.
- Short summaries and next actions.
Recommended implementation:
1. Extend Nightly Obsidian Vault Sync or create a separate evening digest workflow.
2. Pull n8n Failure Digest output, cron/timer health, and Obsidian note deltas.
3. Send concise report to Telegram/Discord.
4. Save durable digest note in Obsidian.
### Weekly review
Desired scope:
- Obsidian note changes.
- Calendar summary.
- GitHub/Linear activity.
- Inbox triage rollup.
- Open tasks and decisions.
Recommended implementation:
1. Use n8n weekly trigger.
2. Collect structured inputs from calendar, Obsidian, GitHub/Linear, email summaries.
3. Use Atlas/Hermes or cloud model for final synthesis.
4. Write `Notes/YYYY-MM-DD Weekly Review.md`.
### Obsidian Semantic Index
Desired scope:
- Watch vault changes.
- Chunk changed notes.
- Embed with Ollama on `18807` using `nomic-embed-text`.
- Store vectors locally.
- Enable semantic search / RAG for Atlas.
Recommended implementation options:
1. Prefer Hermes `rag-search`/local ChromaDB if already available and stable.
2. If n8n owns the trigger, have n8n call a local indexing webhook/script rather than implementing vector DB logic entirely in n8n.
3. Use file-change polling if native file watch is unreliable in Docker/virtiofs.
Open questions:
- Which vector store should be canonical: Hermes rag-search ChromaDB, a separate Chroma instance, SQLite vector extension, or another local store?
- Should n8n trigger indexing, or should Atlas/Hermes own indexing and n8n only notify?
### Personal data routing
Desired scope:
- Gmail/IMAP items to Obsidian.
- Telegram/Discord messages to notes or task queues.
- Calendar events to prep docs.
- YouTube/RSS/blog links to summaries.
Current status:
- Email and calendar pieces exist.
- URL capture exists.
- Telegram/Discord message-to-note routing is not fully built.
- RSS/blog monitoring is not fully built in n8n.
Recommended implementation:
1. Define routing taxonomy and destination folders/tags.
2. Implement one source at a time.
3. Avoid dumping noisy raw messages into Obsidian; use classification gates.
## Recommended next order of work
1. Fix Gmail Trigger warning.
- Goal: no startup warning and confirmed Gmail ingestion.
- Fallback: replace trigger with scheduled polling.
2. Add Docker health-state enrichment to Swarm Health Watchdog.
- Prefer safe host-side helper over raw Docker socket mount.
- Include last Docker health state and suggested fix in alerts.
3. Build Morning Brief v1.
- Start small: calendar + weather + email highlights + swarm/n8n health.
- Deliver to Telegram/Discord and save to Obsidian.
4. Extend Evening Digest.
- Use n8n Failure Digest plus workflow/timer run summary.
- Include failed jobs and next recommended action.
5. Implement Obsidian Semantic Index.
- Decide canonical vector store first.
- Use Ollama embeddings on `18807`.
- Add incremental update path.
6. Upgrade Web-to-Notes Capture.
- Add PDF and YouTube transcript support.
- Add source metadata and claim extraction.
7. Upgrade Voice Memo Pipeline.
- Add native Telegram/Discord voice ingestion.
- Add optional Kokoro audio summary.
8. Define webhook action bus catalog.
- Document stable endpoints and schemas.
- Add `process_url`, `summarize_pdf`, `add_reminder`, `sync_vault`, `run_health_check`.
## Verification commands
Run from host:
```bash
cd /home/will/lab/swarm
make status
curl -fsS --max-time 3 http://127.0.0.1:18808/healthz
docker inspect --format '{{.Name}} {{.State.Status}} {{if .State.Health}}{{.State.Health.Status}}{{else}}no-health{{end}} restarts={{.RestartCount}}' n8n-agent
docker exec n8n-agent n8n export:workflow --all --output=/tmp/workflows-verify.json
```
Inspect workflow inventory from SQLite copy:
```bash
tmp=/tmp/n8n-check-$(date +%s)
mkdir -p "$tmp"
docker cp n8n-agent:/home/node/.n8n/database.sqlite "$tmp/database.sqlite"
sqlite3 "$tmp/database.sqlite" "select name, active, isArchived from workflow_entity order by name;"
```
Check logs without printing secrets:
```bash
docker logs n8n-agent --tail 120
```
## Pitfalls
- Do not assume `/healthz` means all workflows are valid. Verify with `n8n export:workflow --all`.
- If SQLite corruption returns, copy DB files first, then recover offline. Do not make blind in-place edits.
- Validate `workflow_entity` JSON-ish fields after SQLite recovery: `nodes`, `connections`, `settings`, `staticData`, `pinData`, `meta`.
- Empty strings in these fields can crash n8n even when SQLite integrity is `ok`.
- From inside n8n, host-published swarm services use Docker bridge gateway `172.19.0.1:<port>`.
- From inside n8n, n8n self-calls should use `127.0.0.1:5678`, not host port `18808`.
- Do not commit DB backups, workflow execution history, secrets, or runtime state.