docs: document swarm infrastructure topology

This commit is contained in:
William Valentin
2026-05-16 12:45:02 -07:00
parent 0ddd09f007
commit 6a79e0e336
3 changed files with 349 additions and 0 deletions
+113
View File
@@ -0,0 +1,113 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Will's Swarm Infrastructure</title>
<style>
:root { color-scheme: dark; --bg:#020617; --panel:#0f172a; --text:#e2e8f0; --muted:#94a3b8; }
body { margin:0; background:var(--bg); color:var(--text); font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, "Liberation Mono", monospace; }
.wrap { max-width: 1320px; margin: 0 auto; padding: 28px; }
.header { display:flex; align-items:center; gap:12px; margin-bottom:18px; }
.dot { width:12px; height:12px; border-radius:50%; background:#34d399; box-shadow:0 0 18px #34d399; animation:pulse 1.8s infinite; }
@keyframes pulse { 0%,100%{opacity:.6; transform:scale(.9)} 50%{opacity:1; transform:scale(1.15)} }
h1 { font-size: 24px; margin:0; letter-spacing:-.02em; }
.sub { color:var(--muted); margin:4px 0 22px; font-size:13px; }
.card { border:1px solid #1e293b; border-radius:16px; background:linear-gradient(180deg, rgba(15,23,42,.95), rgba(2,6,23,.9)); padding:18px; box-shadow:0 24px 80px rgba(0,0,0,.35); }
svg { width:100%; height:auto; display:block; }
.cards { display:grid; grid-template-columns: repeat(3, minmax(0,1fr)); gap:14px; margin-top:16px; }
.info { border:1px solid #1e293b; border-radius:12px; background:#0f172a; padding:14px; }
.info h3 { margin:0 0 8px; font-size:13px; }
.info ul { margin:0; padding-left:0; list-style:none; color:#cbd5e1; font-size:12px; line-height:1.6; }
.footer { color:#64748b; font-size:11px; margin-top:14px; }
@media (max-width: 900px) { .cards { grid-template-columns: 1fr; } }
</style>
</head>
<body>
<div class="wrap">
<div class="header"><div class="dot"></div><div><h1>Will's Swarm Infrastructure</h1><div class="sub">Atlas/Hermes gateway + n8n automation + agentmon monitoring + local AI/search/voice services</div></div></div>
<div class="card">
<svg viewBox="0 0 1280 900" xmlns="http://www.w3.org/2000/svg" role="img" aria-label="Swarm infrastructure architecture diagram">
<defs>
<pattern id="grid" width="40" height="40" patternUnits="userSpaceOnUse"><path d="M 40 0 L 0 0 0 40" fill="none" stroke="#1e293b" stroke-width="0.5"/></pattern>
<marker id="arrow" markerWidth="10" markerHeight="10" refX="8" refY="3" orient="auto" markerUnits="strokeWidth"><path d="M0,0 L0,6 L9,3 z" fill="#38bdf8" /></marker>
<marker id="arrowGreen" markerWidth="10" markerHeight="10" refX="8" refY="3" orient="auto" markerUnits="strokeWidth"><path d="M0,0 L0,6 L9,3 z" fill="#34d399" /></marker>
<marker id="arrowOrange" markerWidth="10" markerHeight="10" refX="8" refY="3" orient="auto" markerUnits="strokeWidth"><path d="M0,0 L0,6 L9,3 z" fill="#fb923c" /></marker>
<marker id="arrowRose" markerWidth="10" markerHeight="10" refX="8" refY="3" orient="auto" markerUnits="strokeWidth"><path d="M0,0 L0,6 L9,3 z" fill="#fb7185" /></marker>
<filter id="glow"><feGaussianBlur stdDeviation="2.5" result="coloredBlur"/><feMerge><feMergeNode in="coloredBlur"/><feMergeNode in="SourceGraphic"/></feMerge></filter>
<style>
.title{font:700 13px monospace; fill:#e2e8f0}.label{font:11px monospace; fill:#cbd5e1}.tiny{font:9px monospace; fill:#94a3b8}.port{font:8px monospace; fill:#64748b}
.edge{fill:none; stroke:#38bdf8; stroke-width:1.8; marker-end:url(#arrow); opacity:.8}.edgeG{fill:none; stroke:#34d399; stroke-width:1.8; marker-end:url(#arrowGreen); opacity:.85}.edgeO{fill:none; stroke:#fb923c; stroke-width:1.8; marker-end:url(#arrowOrange); opacity:.85}.edgeR{fill:none; stroke:#fb7185; stroke-width:1.8; stroke-dasharray:5,4; marker-end:url(#arrowRose); opacity:.85}
</style>
</defs>
<rect width="1280" height="900" fill="#020617"/><rect width="1280" height="900" fill="url(#grid)" opacity="0.7"/>
<!-- arrows behind nodes -->
<path class="edge" d="M140 120 C210 120 210 205 280 205"/>
<path class="edge" d="M140 190 C210 190 210 235 280 235"/>
<path class="edge" d="M140 260 C210 260 210 265 280 265"/>
<path class="edgeG" d="M470 240 C545 240 545 320 620 320"/>
<path class="edgeG" d="M470 240 C545 240 545 455 620 455"/>
<path class="edgeO" d="M820 320 C890 320 890 210 965 210"/>
<path class="edgeO" d="M820 320 C890 320 890 315 965 315"/>
<path class="edgeO" d="M820 320 C890 320 890 420 965 420"/>
<path class="edgeR" d="M820 455 C890 455 890 595 965 595"/>
<path class="edgeR" d="M820 455 C890 455 890 705 965 705"/>
<path class="edgeG" d="M820 455 C890 455 890 790 965 790"/>
<path class="edge" d="M815 635 C900 635 900 650 965 650"/>
<path class="edge" d="M815 695 C900 695 900 735 965 735"/>
<path class="edgeG" d="M625 635 C555 635 555 720 470 720"/>
<path class="edge" d="M470 720 C545 720 545 565 620 565"/>
<!-- boundaries -->
<rect x="250" y="80" width="250" height="260" rx="14" fill="none" stroke="#fbbf24" stroke-width="1.4" stroke-dasharray="8,5" opacity=".75"/>
<text x="265" y="103" class="tiny" fill="#fbbf24">Hermes gateway layer</text>
<rect x="590" y="105" width="260" height="655" rx="14" fill="none" stroke="#fbbf24" stroke-width="1.4" stroke-dasharray="8,5" opacity=".75"/>
<text x="605" y="128" class="tiny" fill="#fbbf24">n8n + agentmon observability</text>
<rect x="935" y="95" width="280" height="760" rx="14" fill="none" stroke="#fbbf24" stroke-width="1.4" stroke-dasharray="8,5" opacity=".75"/>
<text x="950" y="118" class="tiny" fill="#fbbf24">local swarm services</text>
<!-- external channels -->
<g><rect x="30" y="90" width="110" height="58" rx="8" fill="#0f172a"/><rect x="30" y="90" width="110" height="58" rx="8" fill="rgba(30,41,59,.5)" stroke="#94a3b8" stroke-width="1.5"/><text x="50" y="116" class="title">Telegram</text><text x="52" y="134" class="tiny">DM/groups</text></g>
<g><rect x="30" y="160" width="110" height="58" rx="8" fill="#0f172a"/><rect x="30" y="160" width="110" height="58" rx="8" fill="rgba(30,41,59,.5)" stroke="#94a3b8" stroke-width="1.5"/><text x="52" y="186" class="title">Discord</text><text x="48" y="204" class="tiny">#ops-alerts</text></g>
<g><rect x="30" y="230" width="110" height="58" rx="8" fill="#0f172a"/><rect x="30" y="230" width="110" height="58" rx="8" fill="rgba(30,41,59,.5)" stroke="#94a3b8" stroke-width="1.5"/><text x="65" y="256" class="title">Email</text><text x="48" y="274" class="tiny">Gmail IMAP</text></g>
<!-- Hermes -->
<g filter="url(#glow)"><rect x="280" y="180" width="190" height="100" rx="10" fill="#0f172a"/><rect x="280" y="180" width="190" height="100" rx="10" fill="rgba(8,51,68,.4)" stroke="#22d3ee" stroke-width="1.8"/><text x="325" y="213" class="title">Atlas / Hermes</text><text x="310" y="235" class="label">default profile gateway</text><text x="318" y="258" class="tiny">tools • memory • specialists</text></g>
<!-- n8n and agentmon -->
<g><rect x="620" y="280" width="200" height="80" rx="10" fill="#0f172a"/><rect x="620" y="280" width="200" height="80" rx="10" fill="rgba(6,78,59,.4)" stroke="#34d399" stroke-width="1.8"/><text x="705" y="312" text-anchor="middle" class="title">n8n-agent</text><text x="705" y="333" text-anchor="middle" class="tiny">automation workflows</text><text x="705" y="350" text-anchor="middle" class="port">:18808 host / :5678 container</text></g>
<g><rect x="620" y="415" width="200" height="85" rx="10" fill="#0f172a"/><rect x="620" y="415" width="200" height="85" rx="10" fill="rgba(6,78,59,.4)" stroke="#34d399" stroke-width="1.8"/><text x="720" y="445" text-anchor="middle" class="title">agentmon-query</text><text x="720" y="466" text-anchor="middle" class="tiny">aggregate snapshots/API</text><text x="720" y="484" text-anchor="middle" class="port">:8081 /v1/events</text></g>
<g><rect x="620" y="530" width="200" height="210" rx="10" fill="#0f172a"/><rect x="620" y="530" width="200" height="210" rx="10" fill="rgba(251,146,60,.14)" stroke="#fb923c" stroke-width="1.8"/><text x="720" y="560" text-anchor="middle" class="title">agentmon pipeline</text><text x="720" y="590" text-anchor="middle" class="tiny">ingest :8080</text><text x="720" y="615" text-anchor="middle" class="tiny">NATS JetStream</text><text x="720" y="640" text-anchor="middle" class="tiny">event processor</text><text x="720" y="665" text-anchor="middle" class="tiny">Postgres DB</text><text x="720" y="690" text-anchor="middle" class="tiny">web UI :8082</text><text x="720" y="720" text-anchor="middle" class="port">swarm.snapshot + openclaw.snapshot</text></g>
<!-- Local services -->
<g><rect x="965" y="165" width="210" height="80" rx="9" fill="#0f172a"/><rect x="965" y="165" width="210" height="80" rx="9" fill="rgba(6,78,59,.4)" stroke="#34d399" stroke-width="1.6"/><text x="1070" y="195" text-anchor="middle" class="title">LiteLLM</text><text x="1070" y="216" text-anchor="middle" class="tiny">LLM router + DB</text><text x="1070" y="234" text-anchor="middle" class="port">:18804</text></g>
<g><rect x="965" y="275" width="210" height="80" rx="9" fill="#0f172a"/><rect x="965" y="275" width="210" height="80" rx="9" fill="rgba(8,51,68,.4)" stroke="#22d3ee" stroke-width="1.6"/><text x="1070" y="305" text-anchor="middle" class="title">Search</text><text x="1070" y="326" text-anchor="middle" class="tiny">SearXNG + Brave MCP</text><text x="1070" y="344" text-anchor="middle" class="port">:18803 / :18802</text></g>
<g><rect x="965" y="385" width="210" height="80" rx="9" fill="#0f172a"/><rect x="965" y="385" width="210" height="80" rx="9" fill="rgba(8,51,68,.4)" stroke="#22d3ee" stroke-width="1.6"/><text x="1070" y="415" text-anchor="middle" class="title">Voice</text><text x="1070" y="436" text-anchor="middle" class="tiny">Kokoro + Whisper</text><text x="1070" y="454" text-anchor="middle" class="port">:18805 / :18811</text></g>
<g><rect x="965" y="555" width="210" height="80" rx="9" fill="#0f172a"/><rect x="965" y="555" width="210" height="80" rx="9" fill="rgba(76,29,149,.4)" stroke="#a78bfa" stroke-width="1.6"/><text x="1070" y="585" text-anchor="middle" class="title">Docker services</text><text x="1070" y="606" text-anchor="middle" class="tiny">agentmon.monitor=true</text><text x="1070" y="624" text-anchor="middle" class="port">swarm/service snapshots</text></g>
<g><rect x="965" y="665" width="210" height="80" rx="9" fill="#0f172a"/><rect x="965" y="665" width="210" height="80" rx="9" fill="rgba(120,53,15,.3)" stroke="#fbbf24" stroke-width="1.6"/><text x="1070" y="695" text-anchor="middle" class="title">OpenClaw VMs</text><text x="1070" y="716" text-anchor="middle" class="tiny">currently dormant</text><text x="1070" y="734" text-anchor="middle" class="port">openclaw.snapshot</text></g>
<g><rect x="965" y="775" width="210" height="60" rx="9" fill="#0f172a"/><rect x="965" y="775" width="210" height="60" rx="9" fill="rgba(76,29,149,.4)" stroke="#a78bfa" stroke-width="1.6"/><text x="1070" y="802" text-anchor="middle" class="title">Obsidian / RAG</text><text x="1070" y="822" text-anchor="middle" class="port">:27123/:27124 + ChromaDB</text></g>
<!-- host local ai box -->
<g><rect x="280" y="675" width="190" height="100" rx="10" fill="#0f172a"/><rect x="280" y="675" width="190" height="100" rx="10" fill="rgba(76,29,149,.4)" stroke="#a78bfa" stroke-width="1.8"/><text x="375" y="706" text-anchor="middle" class="title">host local AI</text><text x="375" y="730" text-anchor="middle" class="tiny">llama.cpp :18806</text><text x="375" y="752" text-anchor="middle" class="tiny">Ollama embed :18807</text></g>
<!-- legend -->
<g transform="translate(40,820)">
<text class="tiny" fill="#94a3b8">Legend</text>
<rect x="0" y="16" width="14" height="10" fill="rgba(8,51,68,.4)" stroke="#22d3ee"/><text x="22" y="25" class="tiny">Gateway/Search/Voice</text>
<rect x="180" y="16" width="14" height="10" fill="rgba(6,78,59,.4)" stroke="#34d399"/><text x="202" y="25" class="tiny">Automation/API</text>
<rect x="320" y="16" width="14" height="10" fill="rgba(76,29,149,.4)" stroke="#a78bfa"/><text x="342" y="25" class="tiny">Data/AI stores</text>
<rect x="475" y="16" width="14" height="10" fill="rgba(251,146,60,.14)" stroke="#fb923c"/><text x="497" y="25" class="tiny">Event bus/pipeline</text>
<line x1="650" y1="22" x2="700" y2="22" class="edgeR"/><text x="710" y="25" class="tiny">Monitoring flows</text>
</g>
</svg>
</div>
<div class="cards">
<div class="info"><h3>Monitoring model</h3><ul><li>• n8n direct probes critical ports</li><li>• agentmon aggregates Docker/OpenClaw snapshots</li><li>• n8n polls agentmon for stale/degraded state</li></ul></div>
<div class="info"><h3>Operational endpoints</h3><ul><li>• n8n: 127.0.0.1:18808</li><li>• agentmon query/UI: 8081 / 8082</li><li>• local LLM/embed: 18806 / 18807</li></ul></div>
<div class="info"><h3>Source paths</h3><ul><li>• Swarm repo: ~/lab/swarm</li><li>• Agentmon repo: ~/lab/agentmon</li><li>• Workflows: swarm-common/n8n-workflows</li></ul></div>
</div>
<div class="footer">Generated as repo documentation. Open locally in a browser; no JavaScript, all SVG inline.</div>
</div>
</body>
</html>
+228
View File
@@ -0,0 +1,228 @@
# Swarm Infrastructure
This document is the source-of-truth overview for Will's local swarm/agent infrastructure on the `zap` workstation. It focuses on the runtime services that support Atlas/Hermes, n8n automation, local model/search/voice tooling, Obsidian/RAG automation, and the new agentmon monitoring layer.
## High-level topology
```text
Telegram / Discord / Email
|
v
Hermes / Atlas gateway (default profile)
|
+--> local tools and specialist profiles
+--> n8n automation workflows on :18808
n8n automation
|
+--> direct watchdog probes for key service ports
+--> Agentmon Health Watchdog -> agentmon-query :8081
+--> Obsidian, RAG, voice memo, URL capture, digest workflows
agentmon
|
+--> agentmon-swarm-monitor -> Docker labels agentmon.monitor=true
+--> agentmon-openclaw-monitor -> OpenClaw VM snapshots
+--> NATS JetStream -> event processor -> Postgres
+--> query API / UI on :8081 / :8082
local AI/search/voice services
|
+--> LiteLLM :18804
+--> SearXNG :18803
+--> Brave MCP :18802
+--> llama.cpp :18806
+--> Ollama embeddings :18807
+--> Kokoro TTS :18805
+--> Whisper :18811
```
See also: [`swarm-infrastructure.html`](./swarm-infrastructure.html) for a visual architecture diagram.
## Runtime layers
### 1. Messaging and agent gateway
- **Hermes / Atlas default profile** is the production messaging gateway.
- Connected platforms include Telegram, Discord, and email.
- Atlas uses local swarm services where suitable, especially search, local LLMs, embeddings, STT/TTS, n8n, and agentmon.
- Specialist Hermes profiles are available for delegated work, but the default profile remains the stable production gateway.
### 2. n8n automation
Container/service:
- `n8n-agent`
- Host URL: `http://127.0.0.1:18808`
- Container URL: `http://127.0.0.1:5678`
- Compose project: `/home/will/lab/swarm/docker-compose.yaml`
Important workflow source exports live under:
- `swarm-common/n8n-workflows/`
Current health/automation patterns:
- **Swarm Health Watchdog**: direct endpoint checks for search, LLM, voice, n8n, Docker health, etc.
- **Agentmon Health Watchdog**: polls agentmon aggregate snapshots and alerts on stale/degraded monitoring state.
- **RAG and Embedding Health Watchdog**: checks RAG/search/embedding path.
- Obsidian workflows: health/reindex, inbox triage, daily review, URL-to-note, chat summary capture, weekly decision/runbook extraction.
### 3. Agentmon monitoring layer
Repo:
- `/home/will/lab/agentmon`
Compose services:
- `agentmon-ingest` on `:8080` — ingestion gateway, `/healthz`
- `agentmon-query` on `:8081` — query API, `/healthz`, `/v1/events`, `/v1/stats/summary`
- `agentmon-ui` on `:8082` — web UI, `/healthz`
- `agentmon-processor` — NATS to Postgres event processor
- `agentmon-swarm-monitor` — monitors Docker containers labeled `agentmon.monitor=true`
- `agentmon-openclaw-monitor` — emits OpenClaw VM snapshots
- `agentmon-db` — Postgres
- `agentmon-nats` — NATS JetStream
Key query endpoints:
```text
http://127.0.0.1:8080/healthz
http://127.0.0.1:8081/healthz
http://127.0.0.1:8082/healthz
http://127.0.0.1:8081/v1/stats/summary
http://127.0.0.1:8081/v1/events?event_type=swarm.snapshot&limit=1
http://127.0.0.1:8081/v1/events?event_type=swarm.service.snapshot&limit=20
http://127.0.0.1:8081/v1/events?event_type=openclaw.snapshot&limit=3
```
From inside `n8n-agent`, use the Docker bridge gateway:
```text
http://172.19.0.1:8081/v1/events?event_type=swarm.snapshot&limit=1
```
### 4. Local AI, search, and voice services
Docker services:
- `litellm``:18804`, OpenAI-compatible LLM router
- `litellm-db` — Postgres backing LiteLLM
- `searxng``:18803`, local metasearch
- `brave-search``:18802`, Brave Search MCP server
- `kokoro-tts``:18805`, local TTS
- `whisper-server``:18811`, local transcription
- `n8n-agent``:18808`, automation
Host/user services:
- `llama-server.service``:18806`, local llama.cpp OpenAI-compatible LLM
- `ollama.service``:18807`, embeddings API
- `docker-health-endpoint.service``:18809`, read-only container health for n8n
- `obsidian-reindex-endpoint.service``:18810`, Obsidian/RAG reindex trigger
- `url-content-extractor.service``:18812`, YouTube/PDF/web extraction
- `voice-memo-processor.service``:18813`, voice memo processing
- `rag-embedding-health.service``:18814`, RAG/embedding health wrapper
### 5. Obsidian and RAG
Vault:
- `/home/will/lab/swarm/swarm-common/obsidian-vault/will/will-shared-zap`
Local REST API:
- HTTP: `127.0.0.1:27123`
- HTTPS: `127.0.0.1:27124`
RAG/vector store:
- ChromaDB path: `~/.hermes/data/rag-search/chroma/`
- Embeddings backend: Ollama on `:18807`, normally `nomic-embed-text`
## Monitoring model
The monitoring design is intentionally layered:
1. **n8n direct probes** check critical service endpoints and send deduped alerts.
2. **agentmon** continuously observes labeled Docker services and OpenClaw state, then writes snapshots through NATS/Postgres.
3. **n8n Agentmon Health Watchdog** polls agentmon's aggregate state and alerts if the monitoring pipeline itself becomes stale/degraded.
4. **Hermes/Atlas** can inspect both n8n and agentmon when troubleshooting, and can use the same endpoints as part of operational checks.
This means a single process being alive is not enough: the important signal is whether collection, ingestion, processing, storage, query, and alerting are all functioning.
## Agentmon Health Watchdog
Workflow source:
- `swarm-common/n8n-workflows/agentmon-health-watchdog.json`
Installed n8n workflow:
- Name: `Agentmon Health Watchdog`
- ID: `AgentmonHealthWatchdog`
- Schedule: every 5 minutes
Alert conditions:
- `agentmon-ingest`, `agentmon-query`, or `agentmon-ui` `/healthz` fails.
- Latest `swarm.snapshot` is missing.
- Latest `swarm.snapshot` is older than 3 minutes.
- Snapshot issues are non-empty.
- Required agentmon services are missing or not healthy/running:
- `agentmon-ingest`
- `agentmon-query`
- `agentmon-ui`
- `agentmon-processor`
- `agentmon-swarm-monitor`
- `agentmon-db`
- `agentmon-nats`
Deduplication:
- Alert after 2 failed checks.
- Reminder every 6 failed runs.
- Recovery message when state returns healthy.
## Operational quick checks
From the host:
```bash
cd /home/will/lab/swarm
make status
make local-ai-health
curl -fsS http://127.0.0.1:18808/healthz
curl -fsS http://127.0.0.1:8081/healthz
curl -fsS 'http://127.0.0.1:8081/v1/events?event_type=swarm.snapshot&limit=1' | jq .
```
From inside `n8n-agent`:
```bash
docker exec n8n-agent /bin/sh -lc '
wget -qO- -T 5 http://172.19.0.1:8081/healthz
wget -qO- -T 5 "http://172.19.0.1:8081/v1/events?event_type=swarm.snapshot&limit=1" | head -c 500
'
```
Verify n8n workflow activation:
```bash
docker exec -u node n8n-agent n8n export:workflow \
--id=AgentmonHealthWatchdog \
--output=/tmp/agentmon-export.json
docker cp n8n-agent:/tmp/agentmon-export.json /tmp/agentmon-export.json
jq '.[0] | {id,name,active,nodes:(.nodes|length)}' /tmp/agentmon-export.json
```
## Notes and pitfalls
- Do not commit `.env`, decrypted credentials, raw credential exports, or runtime DB files.
- n8n workflow backups can contain sensitive operational data; keep timestamped raw backups untracked unless intentionally sanitized.
- From host, use `127.0.0.1:<host-port>`.
- From `n8n-agent`, use `127.0.0.1:5678` for n8n itself and `172.19.0.1:<host-port>` for host-published swarm services.
- Agentmon `/healthz` only proves the web/API process is alive; pair it with snapshot freshness to prove the monitoring pipeline is flowing.
- OpenClaw is intentionally dormant unless explicitly re-enabled; do not alert on VMs being shut off by default.