docs(npu): update service maps and runbooks

2026-06-04 13:08:18 -07:00
parent 703c1df860
commit 420df812c0
7 changed files with 1151 additions and 0 deletions
@@ -0,0 +1,66 @@
+# Diagram maintenance
+
+Keep infrastructure diagrams current as first-class documentation, not as one-off screenshots.
+
+## Current diagrams
+
+- [`swarm-infrastructure.html`](./swarm-infrastructure.html) — full Atlas/Hermes + n8n + agentmon + local AI/search/voice topology.
+
+## When to update an existing diagram
+
+Update the relevant diagram in the same change set when you change any of these:
+
+- service topology, ports, or container names
+- monitoring or alerting paths
+- n8n workflow architecture
+- Hermes/Atlas routing or gateway responsibilities
+- local AI/search/voice endpoints
+- OpenVINO NPU live/prototype status, ports, or safety gates (`:18810`, `:18816`, `:18817`, `:18818`, `:18819`, `:18820`, optional `:18829`)
+- Obsidian/RAG data flow
+- OpenClaw/VM operational mode
+- ownership/source-of-truth paths for a component
+
+## When to create a new diagram
+
+Create a new focused diagram when the existing overview would become too dense. Good candidates:
+
+- n8n workflow family or alerting-only diagram
+- agentmon internals: collectors → NATS → processor → Postgres → query/UI
+- Obsidian/RAG automation pipeline
+- local AI routing: Hermes/LiteLLM/llama.cpp/Ollama/provider boundaries
+- OpenVINO NPU assistant sidecars, with live baseline and approved/not-live prototype lanes separated
+- messaging/channel routing: Telegram/Discord/email → Hermes/n8n/alerts
+- disaster recovery / backup topology
+
+## Style rules
+
+- Prefer standalone `.html` files with inline SVG so they render offline in any browser.
+- Keep the source file committed alongside the docs; do not rely on generated screenshots as the only artifact.
+- Link diagrams from the nearest README or operational doc.
+- Keep labels operational: service name, port, responsibility, and data direction.
+- Avoid secrets, credential names that imply secret values, private tokens, raw webhook URLs, or sensitive sample payloads.
+- Do not imply live Atlas/Hermes/RAG routing to an OpenVINO NPU prototype unless a reviewed implementation actually enabled it; label approved prototypes as `not live` or `approval required`.
+- If a raw export or live config was used to build the diagram, commit only the sanitized diagram/docs, not the raw sensitive source.
+
+## Verification before committing
+
+```bash
+# Check the files are valid text and do not contain obvious secret markers
+python - <<'PY'
+from pathlib import Path
+for p in Path('docs').glob('*.html'):
+    text = p.read_text()
+    hits = [s for s in ['api_key', 'token', 'password', 'Authorization', 'Bearer ', 'secret'] if s.lower() in text.lower()]
+    print(p, hits)
+PY
+
+# Inspect targeted diff only
+git diff --stat -- docs README.md
+```
+
+After editing diagrams, commit with a docs-focused message, for example:
+
+```bash
+git add docs/*.md docs/*.html README.md
+git commit -m "docs: update swarm infrastructure diagrams"
+```
@@ -0,0 +1,115 @@
+<!doctype html>
+<html lang="en">
+<head>
+  <meta charset="utf-8" />
+  <meta name="viewport" content="width=device-width, initial-scale=1" />
+  <title>Will's Swarm Infrastructure</title>
+  <style>
+    :root { color-scheme: dark; --bg:#020617; --panel:#0f172a; --text:#e2e8f0; --muted:#94a3b8; }
+    body { margin:0; background:var(--bg); color:var(--text); font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, "Liberation Mono", monospace; }
+    .wrap { max-width: 1320px; margin: 0 auto; padding: 28px; }
+    .header { display:flex; align-items:center; gap:12px; margin-bottom:18px; }
+    .dot { width:12px; height:12px; border-radius:50%; background:#34d399; box-shadow:0 0 18px #34d399; animation:pulse 1.8s infinite; }
+    @keyframes pulse { 0%,100%{opacity:.6; transform:scale(.9)} 50%{opacity:1; transform:scale(1.15)} }
+    h1 { font-size: 24px; margin:0; letter-spacing:-.02em; }
+    .sub { color:var(--muted); margin:4px 0 22px; font-size:13px; }
+    .card { border:1px solid #1e293b; border-radius:16px; background:linear-gradient(180deg, rgba(15,23,42,.95), rgba(2,6,23,.9)); padding:18px; box-shadow:0 24px 80px rgba(0,0,0,.35); }
+    svg { width:100%; height:auto; display:block; }
+    .cards { display:grid; grid-template-columns: repeat(3, minmax(0,1fr)); gap:14px; margin-top:16px; }
+    .info { border:1px solid #1e293b; border-radius:12px; background:#0f172a; padding:14px; }
+    .info h3 { margin:0 0 8px; font-size:13px; }
+    .info ul { margin:0; padding-left:0; list-style:none; color:#cbd5e1; font-size:12px; line-height:1.6; }
+    .footer { color:#64748b; font-size:11px; margin-top:14px; }
+    @media (max-width: 900px) { .cards { grid-template-columns: 1fr; } }
+  </style>
+</head>
+<body>
+  <div class="wrap">
+    <div class="header"><div class="dot"></div><div><h1>Will's Swarm Infrastructure</h1><div class="sub">Atlas/Hermes gateway + n8n automation + agentmon monitoring + local AI/search/voice services</div></div></div>
+    <div class="card">
+      <svg viewBox="0 0 1280 980" xmlns="http://www.w3.org/2000/svg" role="img" aria-label="Swarm infrastructure architecture diagram">
+        <defs>
+          <pattern id="grid" width="40" height="40" patternUnits="userSpaceOnUse"><path d="M 40 0 L 0 0 0 40" fill="none" stroke="#1e293b" stroke-width="0.5"/></pattern>
+          <marker id="arrow" markerWidth="10" markerHeight="10" refX="8" refY="3" orient="auto" markerUnits="strokeWidth"><path d="M0,0 L0,6 L9,3 z" fill="#38bdf8" /></marker>
+          <marker id="arrowGreen" markerWidth="10" markerHeight="10" refX="8" refY="3" orient="auto" markerUnits="strokeWidth"><path d="M0,0 L0,6 L9,3 z" fill="#34d399" /></marker>
+          <marker id="arrowOrange" markerWidth="10" markerHeight="10" refX="8" refY="3" orient="auto" markerUnits="strokeWidth"><path d="M0,0 L0,6 L9,3 z" fill="#fb923c" /></marker>
+          <marker id="arrowRose" markerWidth="10" markerHeight="10" refX="8" refY="3" orient="auto" markerUnits="strokeWidth"><path d="M0,0 L0,6 L9,3 z" fill="#fb7185" /></marker>
+          <filter id="glow"><feGaussianBlur stdDeviation="2.5" result="coloredBlur"/><feMerge><feMergeNode in="coloredBlur"/><feMergeNode in="SourceGraphic"/></feMerge></filter>
+          <style>
+            .title{font:700 13px monospace; fill:#e2e8f0}.label{font:11px monospace; fill:#cbd5e1}.tiny{font:9px monospace; fill:#94a3b8}.port{font:8px monospace; fill:#64748b}
+            .edge{fill:none; stroke:#38bdf8; stroke-width:1.8; marker-end:url(#arrow); opacity:.8}.edgeG{fill:none; stroke:#34d399; stroke-width:1.8; marker-end:url(#arrowGreen); opacity:.85}.edgeO{fill:none; stroke:#fb923c; stroke-width:1.8; marker-end:url(#arrowOrange); opacity:.85}.edgeR{fill:none; stroke:#fb7185; stroke-width:1.8; stroke-dasharray:5,4; marker-end:url(#arrowRose); opacity:.85}
+          </style>
+        </defs>
+        <rect width="1280" height="980" fill="#020617"/><rect width="1280" height="980" fill="url(#grid)" opacity="0.7"/>
+
+        <!-- arrows behind nodes -->
+        <path class="edge" d="M140 120 C210 120 210 205 280 205"/>
+        <path class="edge" d="M140 190 C210 190 210 235 280 235"/>
+        <path class="edge" d="M140 260 C210 260 210 265 280 265"/>
+        <path class="edgeG" d="M470 240 C545 240 545 320 620 320"/>
+        <path class="edgeG" d="M470 240 C545 240 545 455 620 455"/>
+        <path class="edgeO" d="M820 320 C890 320 890 210 965 210"/>
+        <path class="edgeO" d="M820 320 C890 320 890 315 965 315"/>
+        <path class="edgeO" d="M820 320 C890 320 890 420 965 420"/>
+        <path class="edgeR" d="M820 455 C890 455 890 595 965 595"/>
+        <path class="edgeR" d="M820 455 C890 455 890 705 965 705"/>
+        <path class="edgeG" d="M820 455 C890 455 890 790 965 790"/>
+        <path class="edge" d="M815 635 C900 635 900 650 965 650"/>
+        <path class="edge" d="M815 695 C900 695 900 735 965 735"/>
+        <path class="edgeG" d="M625 635 C555 635 555 720 470 720"/>
+        <path class="edge" d="M470 720 C545 720 545 565 620 565"/>
+        <path class="edgeR" d="M490 735 C620 735 790 880 965 880"/>
+
+        <!-- boundaries -->
+        <rect x="250" y="80" width="250" height="260" rx="14" fill="none" stroke="#fbbf24" stroke-width="1.4" stroke-dasharray="8,5" opacity=".75"/>
+        <text x="265" y="103" class="tiny" fill="#fbbf24">Hermes gateway layer</text>
+        <rect x="590" y="105" width="260" height="655" rx="14" fill="none" stroke="#fbbf24" stroke-width="1.4" stroke-dasharray="8,5" opacity=".75"/>
+        <text x="605" y="128" class="tiny" fill="#fbbf24">n8n + agentmon observability</text>
+        <rect x="935" y="95" width="280" height="850" rx="14" fill="none" stroke="#fbbf24" stroke-width="1.4" stroke-dasharray="8,5" opacity=".75"/>
+        <text x="950" y="118" class="tiny" fill="#fbbf24">local swarm services</text>
+
+        <!-- external channels -->
+        <g><rect x="30" y="90" width="110" height="58" rx="8" fill="#0f172a"/><rect x="30" y="90" width="110" height="58" rx="8" fill="rgba(30,41,59,.5)" stroke="#94a3b8" stroke-width="1.5"/><text x="50" y="116" class="title">Telegram</text><text x="52" y="134" class="tiny">DM/groups</text></g>
+        <g><rect x="30" y="160" width="110" height="58" rx="8" fill="#0f172a"/><rect x="30" y="160" width="110" height="58" rx="8" fill="rgba(30,41,59,.5)" stroke="#94a3b8" stroke-width="1.5"/><text x="52" y="186" class="title">Discord</text><text x="48" y="204" class="tiny">#ops-alerts</text></g>
+        <g><rect x="30" y="230" width="110" height="58" rx="8" fill="#0f172a"/><rect x="30" y="230" width="110" height="58" rx="8" fill="rgba(30,41,59,.5)" stroke="#94a3b8" stroke-width="1.5"/><text x="65" y="256" class="title">Email</text><text x="48" y="274" class="tiny">Gmail IMAP</text></g>
+
+        <!-- Hermes -->
+        <g filter="url(#glow)"><rect x="280" y="180" width="190" height="100" rx="10" fill="#0f172a"/><rect x="280" y="180" width="190" height="100" rx="10" fill="rgba(8,51,68,.4)" stroke="#22d3ee" stroke-width="1.8"/><text x="325" y="213" class="title">Atlas / Hermes</text><text x="310" y="235" class="label">default profile gateway</text><text x="318" y="258" class="tiny">tools • memory • specialists</text></g>
+
+        <!-- n8n and agentmon -->
+        <g><rect x="620" y="280" width="200" height="80" rx="10" fill="#0f172a"/><rect x="620" y="280" width="200" height="80" rx="10" fill="rgba(6,78,59,.4)" stroke="#34d399" stroke-width="1.8"/><text x="705" y="312" text-anchor="middle" class="title">n8n-agent</text><text x="705" y="333" text-anchor="middle" class="tiny">automation workflows</text><text x="705" y="350" text-anchor="middle" class="port">:18808 host / :5678 container</text></g>
+        <g><rect x="620" y="415" width="200" height="85" rx="10" fill="#0f172a"/><rect x="620" y="415" width="200" height="85" rx="10" fill="rgba(6,78,59,.4)" stroke="#34d399" stroke-width="1.8"/><text x="720" y="445" text-anchor="middle" class="title">agentmon-query</text><text x="720" y="466" text-anchor="middle" class="tiny">aggregate snapshots/API</text><text x="720" y="484" text-anchor="middle" class="port">:8081 /v1/events</text></g>
+        <g><rect x="620" y="530" width="200" height="210" rx="10" fill="#0f172a"/><rect x="620" y="530" width="200" height="210" rx="10" fill="rgba(251,146,60,.14)" stroke="#fb923c" stroke-width="1.8"/><text x="720" y="560" text-anchor="middle" class="title">agentmon pipeline</text><text x="720" y="590" text-anchor="middle" class="tiny">ingest :8080</text><text x="720" y="615" text-anchor="middle" class="tiny">NATS JetStream</text><text x="720" y="640" text-anchor="middle" class="tiny">event processor</text><text x="720" y="665" text-anchor="middle" class="tiny">Postgres DB</text><text x="720" y="690" text-anchor="middle" class="tiny">web UI :8082</text><text x="720" y="720" text-anchor="middle" class="port">swarm.snapshot + openclaw.snapshot</text></g>
+
+        <!-- Local services -->
+        <g><rect x="965" y="165" width="210" height="80" rx="9" fill="#0f172a"/><rect x="965" y="165" width="210" height="80" rx="9" fill="rgba(6,78,59,.4)" stroke="#34d399" stroke-width="1.6"/><text x="1070" y="195" text-anchor="middle" class="title">LiteLLM</text><text x="1070" y="216" text-anchor="middle" class="tiny">LLM router + DB</text><text x="1070" y="234" text-anchor="middle" class="port">:18804</text></g>
+        <g><rect x="965" y="275" width="210" height="80" rx="9" fill="#0f172a"/><rect x="965" y="275" width="210" height="80" rx="9" fill="rgba(8,51,68,.4)" stroke="#22d3ee" stroke-width="1.6"/><text x="1070" y="305" text-anchor="middle" class="title">Search</text><text x="1070" y="326" text-anchor="middle" class="tiny">SearXNG + Brave MCP</text><text x="1070" y="344" text-anchor="middle" class="port">:18803 / :18802</text></g>
+        <g><rect x="965" y="385" width="210" height="80" rx="9" fill="#0f172a"/><rect x="965" y="385" width="210" height="80" rx="9" fill="rgba(8,51,68,.4)" stroke="#22d3ee" stroke-width="1.6"/><text x="1070" y="415" text-anchor="middle" class="title">Voice</text><text x="1070" y="436" text-anchor="middle" class="tiny">Kokoro + Whisper</text><text x="1070" y="454" text-anchor="middle" class="port">:18805 / :18816</text></g>
+        <g><rect x="965" y="555" width="210" height="80" rx="9" fill="#0f172a"/><rect x="965" y="555" width="210" height="80" rx="9" fill="rgba(76,29,149,.4)" stroke="#a78bfa" stroke-width="1.6"/><text x="1070" y="585" text-anchor="middle" class="title">Docker services</text><text x="1070" y="606" text-anchor="middle" class="tiny">agentmon.monitor=true</text><text x="1070" y="624" text-anchor="middle" class="port">swarm/service snapshots</text></g>
+        <g><rect x="965" y="665" width="210" height="80" rx="9" fill="#0f172a"/><rect x="965" y="665" width="210" height="80" rx="9" fill="rgba(120,53,15,.3)" stroke="#fbbf24" stroke-width="1.6"/><text x="1070" y="695" text-anchor="middle" class="title">OpenClaw VMs</text><text x="1070" y="716" text-anchor="middle" class="tiny">currently dormant</text><text x="1070" y="734" text-anchor="middle" class="port">openclaw.snapshot</text></g>
+        <g><rect x="965" y="775" width="210" height="75" rx="9" fill="#0f172a"/><rect x="965" y="775" width="210" height="75" rx="9" fill="rgba(76,29,149,.4)" stroke="#a78bfa" stroke-width="1.6"/><text x="1070" y="802" text-anchor="middle" class="title">Obsidian / RAG</text><text x="1070" y="821" text-anchor="middle" class="tiny">RAG endpoint :18810</text><text x="1070" y="840" text-anchor="middle" class="port">Chroma obsidian_bge_npu</text></g>
+        <g><rect x="965" y="870" width="210" height="80" rx="9" fill="#0f172a"/><rect x="965" y="870" width="210" height="80" rx="9" fill="rgba(244,63,94,.16)" stroke="#fb7185" stroke-width="1.6" stroke-dasharray="6,4"/><text x="1070" y="896" text-anchor="middle" class="title">NPU sidecars</text><text x="1070" y="917" text-anchor="middle" class="tiny">approved prototypes; not live</text><text x="1070" y="936" text-anchor="middle" class="port">:18818/:18819/:18820/:18829</text></g>
+
+        <!-- host local ai box -->
+        <g><rect x="280" y="675" width="210" height="145" rx="10" fill="#0f172a"/><rect x="280" y="675" width="210" height="145" rx="10" fill="rgba(76,29,149,.4)" stroke="#a78bfa" stroke-width="1.8"/><text x="385" y="706" text-anchor="middle" class="title">host local AI</text><text x="385" y="730" text-anchor="middle" class="tiny">llama.cpp :18806</text><text x="385" y="752" text-anchor="middle" class="tiny">Ollama fallback :18807</text><text x="385" y="774" text-anchor="middle" class="tiny">OpenVINO embed :18817 live</text><text x="385" y="797" text-anchor="middle" class="tiny">Whisper NPU :18816 live</text></g>
+
+        <!-- legend -->
+        <g transform="translate(40,910)">
+          <text class="tiny" fill="#94a3b8">Legend</text>
+          <rect x="0" y="16" width="14" height="10" fill="rgba(8,51,68,.4)" stroke="#22d3ee"/><text x="22" y="25" class="tiny">Gateway/Search/Voice</text>
+          <rect x="180" y="16" width="14" height="10" fill="rgba(6,78,59,.4)" stroke="#34d399"/><text x="202" y="25" class="tiny">Automation/API</text>
+          <rect x="320" y="16" width="14" height="10" fill="rgba(76,29,149,.4)" stroke="#a78bfa"/><text x="342" y="25" class="tiny">Data/AI stores</text>
+          <rect x="475" y="16" width="14" height="10" fill="rgba(251,146,60,.14)" stroke="#fb923c"/><text x="497" y="25" class="tiny">Event bus/pipeline</text>
+          <line x1="650" y1="22" x2="700" y2="22" class="edgeR"/><text x="710" y="25" class="tiny">Monitoring / not-live prototype flows</text>
+        </g>
+      </svg>
+    </div>
+    <div class="cards">
+      <div class="info"><h3>Monitoring model</h3><ul><li>• n8n direct probes critical ports</li><li>• agentmon aggregates Docker/OpenClaw snapshots</li><li>• n8n polls agentmon for stale/degraded state</li></ul></div>
+      <div class="info"><h3>Operational endpoints</h3><ul><li>• n8n: 127.0.0.1:18808</li><li>• agentmon query/UI: 8081 / 8082</li><li>• live NPU: RAG 18810, Whisper 18816, embeddings 18817</li><li>• prototypes not live-routed: 18818/18819/18820/18829</li></ul></div>
+      <div class="info"><h3>Source paths</h3><ul><li>• Swarm repo: ~/lab/swarm</li><li>• Agentmon repo: ~/lab/agentmon</li><li>• Workflows: swarm-common/n8n-workflows</li></ul></div>
+    </div>
+    <div class="footer">Generated as repo documentation. Open locally in a browser; no JavaScript, all SVG inline. Dashed red OpenVINO NPU sidecars are approved prototypes only and do not imply live Atlas/Hermes/RAG routing.</div>
+  </div>
+</body>
+</html>
@@ -0,0 +1,250 @@
+# Swarm Infrastructure
+
+This document is the source-of-truth overview for Will's local swarm/agent infrastructure on the `zap` workstation. It focuses on the runtime services that support Atlas/Hermes, n8n automation, local model/search/voice tooling, Obsidian/RAG automation, and the new agentmon monitoring layer.
+
+## High-level topology
+
+```text
+Telegram / Discord / Email
+        |
+        v
+Hermes / Atlas gateway (default profile)
+        |
+        +--> local tools and specialist profiles
+        +--> n8n automation workflows on :18808
+
+n8n automation
+        |
+        +--> direct watchdog probes for key service ports
+        +--> Agentmon Health Watchdog -> agentmon-query :8081
+        +--> Obsidian, RAG, voice memo, URL capture, digest workflows
+
+agentmon
+        |
+        +--> agentmon-swarm-monitor -> Docker labels agentmon.monitor=true
+        +--> agentmon-openclaw-monitor -> OpenClaw VM snapshots
+        +--> NATS JetStream -> event processor -> Postgres
+        +--> query API / UI on :8081 / :8082
+
+local AI/search/voice services
+        |
+        +--> LiteLLM :18804
+        +--> SearXNG :18803
+        +--> Brave MCP :18802
+        +--> llama.cpp :18806
+        +--> Ollama embeddings :18807 (legacy/CPU fallback)
+        +--> OpenVINO NPU embeddings :18817
+        +--> Kokoro TTS :18805
+        +--> Whisper NPU :18816
+        +--> approved/not-live NPU sidecars: reranker :18818, router/classifier :18819, GenAI worker :18820, doc/image triage optional :18829
+```
+
+See also:
+
+- [`swarm-infrastructure.html`](./swarm-infrastructure.html) — visual architecture diagram
+- [`diagram-maintenance.md`](./diagram-maintenance.md) — how to keep diagrams updated and when to create new ones
+
+## Runtime layers
+
+### 1. Messaging and agent gateway
+
+- **Hermes / Atlas default profile** is the production messaging gateway.
+- Connected platforms include Telegram, Discord, and email.
+- Atlas uses local swarm services where suitable, especially search, local LLMs, embeddings, STT/TTS, n8n, and agentmon.
+- Specialist Hermes profiles are available for delegated work, but the default profile remains the stable production gateway.
+
+### 2. n8n automation
+
+Container/service:
+
+- `n8n-agent`
+- Host URL: `http://127.0.0.1:18808`
+- Container URL: `http://127.0.0.1:5678`
+- Compose project: `/home/will/lab/swarm/docker-compose.yaml`
+
+Important workflow source exports live under:
+
+- `swarm-common/n8n-workflows/`
+
+Current health/automation patterns:
+
+- **Swarm Health Watchdog**: direct endpoint checks for search, LLM, voice, n8n, Docker health, etc.
+- **Agentmon Health Watchdog**: polls agentmon aggregate snapshots and alerts on stale/degraded monitoring state.
+- **RAG and Embedding Health Watchdog**: checks RAG/search/embedding path.
+- Obsidian workflows: health/reindex, inbox triage, daily review, URL-to-note, chat summary capture, weekly decision/runbook extraction.
+
+### 3. Agentmon monitoring layer
+
+Repo:
+
+- `/home/will/lab/agentmon`
+
+Compose services:
+
+- `agentmon-ingest` on `:8080` — ingestion gateway, `/healthz`
+- `agentmon-query` on `:8081` — query API, `/healthz`, `/v1/events`, `/v1/stats/summary`
+- `agentmon-ui` on `:8082` — web UI, `/healthz`
+- `agentmon-processor` — NATS to Postgres event processor
+- `agentmon-swarm-monitor` — monitors Docker containers labeled `agentmon.monitor=true`
+- `agentmon-openclaw-monitor` — emits OpenClaw VM snapshots
+- `agentmon-db` — Postgres
+- `agentmon-nats` — NATS JetStream
+
+Key query endpoints:
+
+```text
+http://127.0.0.1:8080/healthz
+http://127.0.0.1:8081/healthz
+http://127.0.0.1:8082/healthz
+http://127.0.0.1:8081/v1/stats/summary
+http://127.0.0.1:8081/v1/events?event_type=swarm.snapshot&limit=1
+http://127.0.0.1:8081/v1/events?event_type=swarm.service.snapshot&limit=20
+http://127.0.0.1:8081/v1/events?event_type=openclaw.snapshot&limit=3
+```
+
+From inside `n8n-agent`, use the Docker bridge gateway:
+
+```text
+http://172.19.0.1:8081/v1/events?event_type=swarm.snapshot&limit=1
+```
+
+### 4. Local AI, search, and voice services
+
+Docker services:
+
+- `litellm` — `:18804`, OpenAI-compatible LLM router
+- `litellm-db` — Postgres backing LiteLLM
+- `searxng` — `:18803`, local metasearch
+- `brave-search` — `:18802`, Brave Search MCP server
+- `kokoro-tts` — `:18805`, local TTS
+- `whisper-server-npu` — `:18816`, OpenVINO NPU local transcription
+- `n8n-agent` — `:18808`, automation
+
+Host/user services:
+
+- `llama-server.service` — `:18806`, local llama.cpp OpenAI-compatible LLM
+- `ollama.service` — `:18807`, legacy/CPU embeddings API fallback
+- `openvino-embeddings.service` — `:18817`, OpenVINO NPU embeddings API (`/v1/embeddings`, `/api/embed`, `/api/embeddings`)
+- `docker-health-endpoint.service` — `:18809`, read-only container health for n8n
+- `obsidian-reindex-endpoint.service` — `:18810`, Obsidian/RAG reindex trigger; default collection `obsidian_bge_npu` using OpenVINO NPU embeddings
+- `url-content-extractor.service` — `:18812`, YouTube/PDF/web extraction
+- `voice-memo-processor.service` — `:18813`, voice memo processing
+- `rag-embedding-health.service` — `:18814`, RAG/embedding health wrapper
+
+Approved but not live-routed OpenVINO NPU sidecars:
+
+| Port | Component | State | Safety boundary |
+| ---: | --- | --- | --- |
+| `18818` | reranker | approved prototype; optional foreground/user-systemd only | request-time only; no Chroma/vector mutation; no live RAG integration unless Will approves |
+| `18819` | router/classifier | approved prototype; dry-run only | no Hermes/Atlas routing, memory writes, service restarts, or outbound messages |
+| `18820` | bounded GenAI worker | approved prototype | background jobs only; not primary Atlas/Hermes model routing |
+| `18829` | document/image triage | CLI-first; optional localhost server | synthetic/non-private smoke data only; no private directory processing; NPU stage is embeddings via `:18817` |
+
+These sidecars must bind to `127.0.0.1` by default, must not be enabled persistently or wired into live Atlas/Hermes/RAG paths without explicit Will approval, and any NPU claim requires a positive `/sys/class/accel/accel0/device/npu_busy_time_us` delta before/after inference. HTTP 200 alone is not proof.
+
+### 5. Obsidian and RAG
+
+Vault:
+
+- `/home/will/lab/swarm/swarm-common/obsidian-vault/will/will-shared-zap`
+
+Local REST API:
+
+- HTTP: `127.0.0.1:27123`
+- HTTPS: `127.0.0.1:27124`
+
+RAG/vector store:
+
+- ChromaDB path: `~/.hermes/data/rag-search/chroma/`
+- Reindex state/progress: active BGE/NPU state in `~/.hermes/data/rag-search/obsidian_bge_npu_index_state.json` and `obsidian_bge_npu_reindex_progress.json`; legacy Ollama state in `obsidian_index_state.json` remains for comparison/fallback.
+- Active RAG query/reindex embedding backend: OpenVINO NPU embeddings service on `:18817`, currently `bge-base-en-v1.5-int8-ov`, collection `obsidian_bge_npu`.
+- Legacy comparison/fallback collection: `obsidian`, built with Ollama on `:18807` using `nomic-embed-text`.
+- Reindex endpoint: `POST :18810/reindex` for incremental updates, `POST :18810/reindex?full=true` for full semantic rebuilds, `GET :18810/semantic-health` to verify vectors plus a search smoke test.
+
+## Monitoring model
+
+The monitoring design is intentionally layered:
+
+1. **n8n direct probes** check critical service endpoints and send deduped alerts.
+2. **agentmon** continuously observes labeled Docker services and OpenClaw state, then writes snapshots through NATS/Postgres.
+3. **n8n Agentmon Health Watchdog** polls agentmon's aggregate state and alerts if the monitoring pipeline itself becomes stale/degraded.
+4. **Hermes/Atlas** can inspect both n8n and agentmon when troubleshooting, and can use the same endpoints as part of operational checks.
+
+This means a single process being alive is not enough: the important signal is whether collection, ingestion, processing, storage, query, and alerting are all functioning.
+
+## Agentmon Health Watchdog
+
+Workflow source:
+
+- `swarm-common/n8n-workflows/agentmon-health-watchdog.json`
+
+Installed n8n workflow:
+
+- Name: `Agentmon Health Watchdog`
+- ID: `AgentmonHealthWatchdog`
+- Schedule: every 5 minutes
+
+Alert conditions:
+
+- `agentmon-ingest`, `agentmon-query`, or `agentmon-ui` `/healthz` fails.
+- Latest `swarm.snapshot` is missing.
+- Latest `swarm.snapshot` is older than 3 minutes.
+- Snapshot issues are non-empty.
+- Required agentmon services are missing or not healthy/running:
+  - `agentmon-ingest`
+  - `agentmon-query`
+  - `agentmon-ui`
+  - `agentmon-processor`
+  - `agentmon-swarm-monitor`
+  - `agentmon-db`
+  - `agentmon-nats`
+
+Deduplication:
+
+- Alert after 2 failed checks.
+- Reminder every 6 failed runs.
+- Recovery message when state returns healthy.
+
+## Operational quick checks
+
+From the host:
+
+```bash
+cd /home/will/lab/swarm
+make status
+make local-ai-health
+./scripts/npu-service-health.sh  # read-only; includes sysfs busy-time proof for :18817
+curl -fsS http://127.0.0.1:18808/healthz
+curl -fsS http://127.0.0.1:8081/healthz
+curl -fsS 'http://127.0.0.1:8081/v1/events?event_type=swarm.snapshot&limit=1' | jq .
+```
+
+From inside `n8n-agent`:
+
+```bash
+docker exec n8n-agent /bin/sh -lc '
+  wget -qO- -T 5 http://172.19.0.1:8081/healthz
+  wget -qO- -T 5 "http://172.19.0.1:8081/v1/events?event_type=swarm.snapshot&limit=1" | head -c 500
+'
+```
+
+Verify n8n workflow activation:
+
+```bash
+docker exec -u node n8n-agent n8n export:workflow \
+  --id=AgentmonHealthWatchdog \
+  --output=/tmp/agentmon-export.json
+
+docker cp n8n-agent:/tmp/agentmon-export.json /tmp/agentmon-export.json
+jq '.[0] | {id,name,active,nodes:(.nodes|length)}' /tmp/agentmon-export.json
+```
+
+## Notes and pitfalls
+
+- Do not commit `.env`, decrypted credentials, raw credential exports, or runtime DB files.
+- n8n workflow backups can contain sensitive operational data; keep timestamped raw backups untracked unless intentionally sanitized.
+- From host, use `127.0.0.1:<host-port>`.
+- From `n8n-agent`, use `127.0.0.1:5678` for n8n itself and `172.19.0.1:<host-port>` for host-published swarm services.
+- Agentmon `/healthz` only proves the web/API process is alive; pair it with snapshot freshness to prove the monitoring pipeline is flowing.
+- OpenClaw is intentionally dormant unless explicitly re-enabled; do not alert on VMs being shut off by default.
+- OpenVINO NPU sidecars on `:18818`, `:18819`, `:18820`, and optional `:18829` are prototypes/not-live unless a later approved change installs and routes them. Do not draw live Atlas/Hermes/RAG arrows to them in diagrams until that approval and implementation actually exist.